Zack's Kernel News

Zack's Kernel News

Article from Issue 186/2016
Author(s):

Chronicler Zack Brown reports on the latest news, views, dilemmas, and developments within the Linux kernel community.

Software RAID Maintainership

Neil Brown has stepped down as the software RAID maintainer after 15 years at the post. He explained that he'd been losing interest in the project lately and had not been able to attract any potential replacements. As a result, he decided not to push too hard for a clean handoff, but instead to simply step down and allow the vacuum to be filled by whoever ends up having the greatest need to fill it.

In terms of how he envisioned software RAID maintainership in the future, Neil said, "I think it would be great if a (small) team formed to continue to oversee md [Multiple Device driver] rather than just a single individual (or maybe the dm [Device Mapper] team could extend to include md??). If I had managed to be part of a team rather than "going it alone" for so long, I might feel less tired of the whole thing now." And he added:

A few people have been mentioned to me in earlier less-public conversations. Any of them may well be suitable, but I would rather they named themselves if interested.

So, I'm hoping to get one or more volunteers to be maintainer:

  • to gather and manage patches and outstanding issues,
  • to review patches or get them reviewed
  • to follow up bug reports and get them resolved
  • to feed patches upstream, maybe directly to Linus, maybe through some other maintainer, depending on what relationships already exist or can be formed,
  • to guide the longer term direction (saying "no" is important sometimes),
  • to care, but also to be aware that maintainership takes real effort and time, as does anything that is really worthwhile.

This all applies to mdadm as well as md (except you would ultimately *be* upstream for mdadm, not needing to send it anywhere). Even if a clear team doesn't form, it would be great if different people maintained mdadm and md.

One part of the job that I have put a lot of time in to is following the linux-raid@vger.kernel.org list and providing support. This makes people feel good about md and so more adventurous in using it. Consequently, I tend to hear about bugs and usability issues nice and early (well before paying customers hit them in most cases), and that is a big win.

In recent times, I've been doing less of this and have been absolutely thrilled that the gap has been more than filled by other very competent community members. Not developers in particular, but a number of md users have been providing excellent support. I'd particularly like to highlight Phil Turmel who is very forthcoming with excellent advice, but he is certainly not the only one who deserves a lot of thanks. So "Thank you" to everyone who answers questions on linux-raid.

This would be a good place for any future maintainer to hang out to receive wisdom as well as to provide support.

I will still be around. I can certainly help out in some sort of mentor role, and can probably be convinced to review patches and comment on designs. But I really want to head towards spending less time on md (there are so many other interesting things to learn about).

So: if anyone is interested – please announce yourself, ask questions and start doing things. I have no clear idea about how a transition will happen. That is really up to you (plural). Take the bull by the horns and start *being* a maintainer(team). I won't get in your way, and I'll help where I can.

There were a lot of replies from people who were either interested in joining a maintainer team or were interested in bowing out gracefully. Phil Turmel hinted that he might join in, though he said his kernel skills might need polishing. Shaohua Li said that he'd be interested in joining the team and had support from his employer, Facebook. Jonathan Brassow suggested that a bunch of Red Hat employees might be interested in joining a maintainer team. Jes Sorensen also offered to be a part of the team and also endorsed Shaohua as an important potential teammate. Artur Paszkiewicz also volunteered to join the team.

Neil was very happy to see this level of interest and offered some parting advice: "The first question is where do you send your patches to get the appropriate review and upstream acceptance. Alasdair or Mike (DM), Jens (Block), Andrew Morton (anything), and Linus (everything) are all defensible choices for upstreaming (I've submitted through Andrew in the past, but through Linus exclusively once I figured out git). That is really something you and they would need to negotiate though."

And, he added, "If you have specific questions about anything (md internals, git workflows, whatever) do feel free to ask. I am trying to write some comprehensive documentation on the internals of MD – though it is slow going. Once that is mostly done, I'll make it available and it should at least serve as a good structure in which to place answers to further questions." Finally, Neil said, "I plan to submit a pull request to Linus for the 4.5 merge window and then stop queuing patches." And, thereafter, he planned to "only look at other patches if they already have an independent 'Reviewed-by'."

In a concluding post, Neil announced, "Jes Serensen will be coordinating mdadm (once I make a release … tomorrow?) and Shaohua Li will be looking after the kernel/md side. … Only one person (at a time) can queue patches, but several can collaborate at development and support and bug fixing and testing." Jes then announced the creation of a new mdadm Git repository [1].

Kernel Documentation System Overhaul

Jonathan Corbet posted a set of patches – intended only as proof-of-concept so far – to convert the Linux kernel to use AsciiDoc for its documentation, instead of DocBook. This would make it easier for contributors, who could use a plain-text markup language instead of XML. It would also allow the makefile to produce HTML, man pages, PDFs, and a lot of other documentation formats. Jonathan posted a link to the Git repository [2] for anyone wanting to tinker around with it.

Jani Nikula announced that he'd been doing some work that overlapped with Jonathan's (Jonathan's tree actually included a patch from Jani, which was the impetus for the whole project). Jani said, "I first took roughly the same approach as you did. I was really impressed with the speed and the beauty of the produced HTML. The trouble is, neither asciidoc nor asciidoctor can produce chunked (split to several pages) HTML directly. This is a showstopper for the gpu document which turns into 1.3MB of HTML, which looks pretty but is a pain to navigate. To do chunked output, you have to output DocBook and handle that like we do now. So while I would like to have asciidoc generate HTML directly for speed and beauty, I ended up going the asciidoc to DocBook path. The upside is all the output formats are supported."

Jani added, "One significant difference between our approaches is that I ditched docproc out of the equation. Instead of having the docproc ! directives in the asciidoc, I opted for using asciidoc's native include macro, with specially crafted filename suffixes to specify what parts of the source to include. Those files are then generated as intermediate asciidoc files using kernel-doc, with dependencies set right. There's a bunch of scripting involved, but it's pretty straightforward."

Daniel Vetter voiced some concern about the need to parallelize doc creation. The sheer size of Linux kernel documentation made this an important issue and related to Jani's consideration of "chunked" output.

Jani replied: "asciidoc->HTML is about as fast as asciidoc->XML, and with parallel kernel-doc, it really is fast. Sadly, the XML- HTML part still takes forever.

"I just want to emphasize that we can get parallel kernel-doc with either pipeline. It is also possible to enable both pipelines, i.e. have a fast path HTML generation with few external dependencies and the Swiss army knife slow path with XML."

Jonathan was excited to collaborate with Jani, saying, "Overlapping work is just how this kernel thing works :)" He proceeded to look over Jani's work so far, and said:

I would *really* like to get the XML step out of the processing path if possible. It adds complexity, makes it harder for others to build the docs, makes things more fragile, and slows it all down. It seems to me that it should be possible to do that.

The issues, it seems, are splitting the output files and format support. The latter isn't really an issue, I don't think; there are tools to do all kinds of format conversions. The only one that's even slightly weird is man, and kernel-doc already has some (unused, I think) provisions for doing that. We could generate man pages directly without much pain.

For HTML-page splitting, we can see if the tools can help us, consider splitting the template files, or do the splitting in a post-processing step. Docproc (or whatever replaces it) could also maybe do that work. It doesn't seem to me something that should force the inclusion of an entire XML-based processing step.

Jonathan added:

I have some thoughts on the whole thing …

  • I would like to format directly to HTML if at all possible. It seems it should be possible to get a table of contents into the files, and the feedback I got was that a TOC would be enough for navigation – it would not be necessary to split the files at that point. We might still want to try to figure that out too, though. In any case, this isn't a show stopper, in that we can change it anytime if a better way shows up. But I'd like to have it in mind.
  • Asciidoc templates and processing should happen in a new directory (perhaps imaginatively called "asciidoc"); having them in a directory called "DocBook" seems a little weird. More importantly, though, I'd like to separate them out as a fresh start, and not mess with the existing DocBook templates until we decide we don't need them anymore. If we could end up with a cleaner, simpler makefile in the process, that would be a bonus.
  • I'm not sold on the new inclusion mechanism. Creating thousands of little files and tracking them for dependencies and such doesn't seem like a simplification or a path toward better performance. I would like to at least consider keeping the direct-from-source inclusion.
  • Insisting on EXPORT_SYMBOL being in the same file doesn't seem like it's going to work for now; that could maybe change after Al's work goes in, which could be fairly soon.

Jonathan asked Jani, "Do you think you can find some time over the next month for this? I'll try to shake loose some time too, but, well, $EXCUSES … ."

Daniel advocated a slow approach, converting the front-line documentation to asciidoc but having the makefile produce DocBook XML, which could then use the existing toolchain to produce the final output. He though that once this was in the source tree, they could gradually remove the DocBook dependencies in later kernel releases.

Jani wanted to get that issue nailed down right away, before deciding on any further course of action. He asked:

Is the aim to have asciidoc->HTML only or dual asciidoc->HTML and asciidoc->XML->whatever? Or independent asciidoc->HTML first, with the existing DocBook on the side until everything's converted? Something else?

Direct asciidoc->HTML has the problem I mentioned that there is no chunked output. If the source is big (as-is or via asciidoc includes) the output is big. The current gpu.tmpl turned way too big. We could alleviate that by splitting the source documents into smaller pieces (in gpu.tmpl case, it's desirable no matter what), and tying them together via cross-references and TOC rather than asciidoc includes.

The problem with this, in turn, is that I don't really know how automatic cross-referencing between kernel-doc comments would turn out then (e.g., i915 kernel-doc references a symbol in drm core kernel-doc after gpu.tmpl split) as asciidoc would process the files independently. A kernel-doc comment writer shouldn't have to know which document the referenced symbol is in … . We could do post-processing I guess, but I'd really like to get rid of the homebrew aspects here.

Is it acceptable to have dead links when referencing symbols outside of the document in question, for the time being, until someone figures out a nice way to do this?

Jani added, "in my dream world, you could have asciidoc files anywhere in the Documentation tree, with a Makefile per directory identifying which ones should be processed as asciidoc. I might even name them all .txt, and you wouldn't have to rename existing 'almost markup' plain text files to have them processed, just fix the markup and update the Makefile. (FWIW asciidoc suggests .txt extension, though asciidoctor suggests .adoc or .asciidoc.) I think this would better promote a gradual transition to lightweight markup, with easier to review patches."

Jonathan said that asciidoc-to-HTML alone wasn't viable, since people wanted other formats. He said that he personally would love to see epub, and he knew of others who preferred man pages. He suspected someone would like to see PDF output, though he doubted anyone had successfully generated kernel docs in PDF format for quite some time.

In terms of splitting the files up, Jonathan suggested "generate a TOC and use CSS to place it correctly," which was an idea that had come up in private conversation and might alleviate the need for chunked output.

Keith Packard and Jonathan both felt that Daniel's slower, short-term approach of having asciidoc translate to DocBook was not their preference. Keith felt it was important to reduce the number of links on the toolchain.

Keith also took a deeper look at asciidoc and reported:

It is a stream processor with a stack. The output is generated with some simple templates, one for each back end. Here's the xhtml1.1 template for section level sections: (sect1 in the .conf file):

[sect1]
<div class="sect1{style? {style}}{role? {role}}">
<h2{id? id="{id}"}>{numbered?{sectnum} }{title}</h2>
<div class="sectionbody">
|
</div>
</div>

The contents of the section get inserted at the |; it's nesting, so [sect2] bits would get expanded while being processed.

Each asciidoc back end has dramatically different functionality. It's pretty clear to me that the 'docbook' back end has the best support for larger documents as that provides 'book-scale' processing bits. I've recently written a book in asciidoc using the docbook back end, and the html and pdf results are quite comparable. Using the html back end from asciidoc yields a significantly different result.

I think it should be pretty easy to hack asciidoc to add diversions to hold TOC contents while generating the rest of the doc and then replay the diversion into the final document.

Keith went on to say, "At the end of the document, we'd have some way of wrapping the diversion in suitable additional bits to complete the TOC, which would then be formatted by CSS.

"This same technique could be used to create lists of figures and tables.

"The goal would be to create an html document which could be used without javascript, and that would work without css as well."

Jani was reluctant to take Keith's ideas too far, however. He said, "One of the chief complaints with the current pipeline (and some of the proposals) has been the need to install lots of tools with lots of dependencies. I would like to avoid the need to install bleeding-edge tools and stick to what's already widely available in distros. Thus, I would like to avoid hacking asciidoc for our needs."

Keith agreed with that, but he said "that means using docbook for now; the native html output from asciidoc is simply not usable for anything more complicated than a short web page. However, getting ready to collapse the pipeline by eliminating docbook seems like a good medium-term goal." Meanwhile, Keith did hack up the asciidoc sources somewhat and sent his patches to the upstream maintainers.

The discussion ended there, but the kernel folks are in the process of descending on a set of tools and wrangling them into the form they want. This would resemble the relationship between the kernel and GCC, GDB, and libc, with cross-over contributions, and the occasional controversy as the kernel's needs fail to overlap with the goals of the tools' native developers.

The Author

The Linux kernel mailing list comprises the core of Linux development activities. Traffic volumes are immense, often reaching 10,000 messages in a week, and keeping up to date with the entire scope of development is a virtually impossible task for one person. One of the few brave souls to take on this task is Zack Brown.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Asciidoctor

    The popular AsciiDoc documentation system still has a lot to offer, but more experienced users should check out Asciidoctor, which has some additional new features.

  • AsciiDoc

    AsciiDoc syntax along with its eponymous command lets users create a text document with unobtrusive markup and convert it to a variety of output formats.

  • Command Line – unoconv

    A hidden utility in the LibreOffice toolbox, unoconv offers a wide array of import and export filter options for use at the command line.

  • Introduction

    This month in Linux Voice.

  • Kernel News

    Zack Brown looks at improving memory management, simplifying(ish) the Kernel Build System, and detecting firmware crashes.

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News