We are pleased to announce the availability of a new mailing list service running under the new lists.linux.dev domain. The goal of this deployment is to offer a subscription service that:

• prioritizes mail delivery to public-inbox archives available via lore.kernel.org
• conforms to DMARC requirements to ensure subscriber delivery
• makes minimal changes to email headers and no changes to the message body content for the purposes of preserving patch attestation

If you would like to host a Linux development mailing list on this platform, please see further details on the subspace.kernel.org site.

If you are a developer located around Beijing, or if your connection to Beijing is faster and more reliable than to locations outside of China, then you may benefit from the new git.kernel.org mirror kindly provided by Code Aurora Forum at https://kernel.source.codeaurora.cn/. This is a full mirror that is updated just as frequently as other git.kernel.org nodes (in fact, it is managed by the same team as the rest of kernel.org infrastructure, since CAF is part of Linux Foundation IT projects).

To start using the Beijing mirror, simply clone from that location or add a separate remote to your existing checkouts, e.g.:

git remote add beijing git://kernel.source.codeaurora.cn/pub/scm/.../linux.git
git fetch beijing master

You may also use http:// and https:// protocols if that makes it easier behind corporate firewalls.

We are trialing out a new feature that can send you a notification when the patches you send to the LKML are applied to linux-next or to the mainline git trees. If you are interested in trying it out, here are the details:

• The patches must be sent to the LKML (linux-kernel@vger.kernel.org).
• One of the cc's must be notify@kernel.org (Bcc will not work).
• Alternatively, there should be a "X-Patchwork-Bot: notify" email header.
• The patches must not have been modified by the maintainer(s).
• All patches in the series must have been applied, not just some of them.

The last two points are important, because if there are changes between the content of the patch as it was first sent to the mailing list, and how it looks like by the time it is applied to linux-next or mainline, the bot will not be able to recognize it as the same patch. Similarly, for series of multiple patches, the bot must be able to successfully match all patches in the series in order for the notification to go out.

If you are using git-format-patch, it is best to add the special header instead of using the Cc notification address, so as to avoid any unnecessary email traffic:

--add-header="X-Patchwork-Bot: notify"

You should receive one notification email per each patch series, so if you send a series of 20 patches, you will get a single email in the form of a reply to the cover letter, or to the first patch in the series. The notification will be sent directly to you, ignoring any other addresses in the Cc field.

The bot uses our LKML patchwork instance to perform matching and tracking, and the source code for the bot is also available if you would like to suggest improvements.

You may access the archives of many Linux development mailing lists on lore.kernel.org. Most of them include a full archive of messages going back several decades.

• listing of currently hosted archives

If you would like to suggest another kernel development mailing list to be included in this list, please follow the instructions on the following wiki page:

• Adding list archives to lore.kernel.org

[lore.kernel.org]
site = https://lore.kernel.org
manifest = https://lore.kernel.org/manifest.js.gz
toplevel = /path/to/your/local/folder
mymanifest = /path/to/your/local/folder/manifest.js.gz
pull_threads = 4

We'd like to announce several small changes to the way Linux tarballs are produced.

If you are in charge of CI infrastructure that needs to perform frequent full clones of kernel trees from git.kernel.org, we strongly recommend that you use the git bundles we provide instead of performing a full clone directly from git repositories.

It is better for you, because downloading the bundle from CDN is probably going to be much faster for you than cloning from our frontends due to the CDN being more local. You can even copy the bundle to a fileserver on your local infrastructure and save a lot of repeated external traffic.

It is better for us, because if you first clone from the bundle, you only need to fetch a handful of newer objects directly from git.kernel.org frontends. This not only uses an order of magnitude less bandwidth, but also results in a much smaller memory footprint on our systems -- git daemon needs a lot of RAM when serving full clones of linux repositories.

Here is a simple script that will help you automate the process of first downloading the git bundle and then fetching the newer objects:

• linux-bundle-clone

Thank you for helping us keep our systems fast and accessible to all.

The Linux Foundation IT team has been working to improve the code integrity of git repositories hosted at kernel.org by promoting the use of PGP-signed git tags and commits. Doing so allows anyone to easily verify that git repositories have not been altered or tampered with no matter from which worldwide mirror they may have been cloned. If the digital signature on your cloned repository matches the PGP key belonging to Linus Torvalds or any other maintainer, then you can be assured that what you have on your computer is the exact replica of the kernel code without any omissions or additions.

To help promote the use of PGP signatures in Linux kernel development, we now offer a detailed guide within the kernel documentation tree:

• Kernel Maintainer PGP Guide

Further, we are happy to announce a new special program sponsored by The Linux Foundation in partnership with Nitrokey -- the developer and manufacturer of smartcard-compatible digital tokens capable of storing private keys and performing PGP operations on-chip. Under this program, any developer who is listed as a maintainer in the MAINTAINERS file, or who has a kernel.org account can qualify for a free digital token to help improve the security of their PGP keys. The cost of the device, including any taxes, shipping and handling will be covered by The Linux Foundation.

To participate in this program, please access the special store front on the Nitrokey website:

• Nitrokey Start for Kernel Developers

As you may be aware, starting with 4.12-rc1 Linus will no longer provide signed tarballs and patches for pre-release ("-rc") kernels. Reasons for this are multiple, but largely this is because people who are most interested in pre-release tags -- kernel developers -- do not rely on patches and tarballs to do their work.

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
git verify-tag v4.12-rc1
git archive --format=tar.gz --prefix=linux-4.12-rc1/ \
  -o linux-4.12-rc1.tar.gz v4.12-rc1

wget https://git.kernel.org/torvalds/t/linux-4.12-rc1.tar.gz

wget -O patch-4.12-rc1 https://git.kernel.org/torvalds/p/v4.12-rc1/v4.11

wget -O patch-4.12-rc1.gz --header="accept-encoding: gzip" \
  https://git.kernel.org/torvalds/p/v4.12-rc1/v4.11

The XZ tarballs for the following kernel releases did not initially pass signature verification due to benign changes to the tarball structure done by the pixz compression tool:

• 4.11.1
• 4.10.16
• 4.9.28
• 4.4.68

These changes would have resulted in GPG returning "Bad Signature" if you tried to verify their integrity. Once we identified the problem, we generated new XZ tarballs without tar header modifications and now they should all pass PGP signature verification.

We preserved the original .xz tarballs as -badsig files in the archives in case you wanted to verify that there was nothing malicious in them, merely tar header changes. You can find them in the same v4.x directory:

• https://www.kernel.org/pub/linux/kernel/v4.x/

Our apologies for this problem and thanks to Brad Spengler and everyone else who alerted us about this issue.

We are extremely happy to announce that Packet has graciously donated the new hardware systems providing read-only public access to the kernel.org git repositories and the public website (git.kernel.org and www.kernel.org, respectively). We have avoided using cloud providers in the past due to security implications of sharing hypervisor memory with external parties, but Packet's hardware-based single-tenant approach satisfies our security requirements while taking over the burden of setting up and managing the physical hardware in multiple worldwide datacenters.

As of March 11, 2017, the four new public frontends are located in the following geographical locations:

• San Jose, California, USA
• Parsippany, New Jersey, USA
• Amsterdam, Netherlands
• Tokyo, Japan

We have changed our DNS configuration to support GeoDNS, so your requests should be routed to the frontend nearest to you.

Each Packet-hosted system is significantly more powerful than our previous generation frontends and have triple the amount of available RAM, so they should be a lot more responsive even when a lot of people are cloning linux.git simultaneously.

Our special thanks to the following organizations who have graciously donated hosting for the previous incarnation of kernel.org frontends:

• Internet Systems Consortium
• Vexxhost
• Tizen

If you notice any problems with the new systems, please email helpdesk@kernel.org.

Those of you who have been around for a while may remember a time when you used to be able to mount kernel.org directly as a partition on your system using NFS (or even SMB/CIFS). The Wayback Machine shows that this was still advertised some time in January 1998, but was removed by the time the December 1998 copy was made.

Let's face it -- while kinda neat and convenient, offering a public NFS/CIFS server was a Pretty Bad Idea, not only because both these protocols are pretty terrible over high latency connections, but also because of important security implications.

Well, 19 years later we're thinking it's time to terminate another service that has important protocol and security implications -- our FTP servers. Our decision is driven by the following considerations:

• The protocol is inefficient and requires adding awkward kludges to firewalls and load-balancing daemons
• FTP servers have no support for caching or accelerators, which has significant performance impacts
• Most software implementations have stagnated and see infrequent updates

All kernel.org FTP services will be shut down by the end of this year. In hopes to minimise the potential disruption, we will be doing it in two stages:

1. ftp://ftp.kernel.org/ service will be terminated on March 1, 2017
2. ftp://mirrors.kernel.org/ service will be terminated on December 1, 2017

If you have any concerns, please feel free to contact ftpadmin@kernel.org (ah, the irony).

If your browser alerted you that the site certificates have changed, that would be because we replaced our StartCOM, Ltd certificates with those offered by our DNS registrar, Gandi. We are very thankful to Gandi for this opportunity.

A common question is why we aren't using the certificates offered by the Let's Encrypt project, and the answer is that there are several technical hurdles (on our end) that currently make it complicated. Once we resolve them, we will most likely switch to using certificates issued by our fellow Linux Foundation project.

If you find yourself on an unreliable Internet connection and need to perform a fresh clone of Linux.git, you may find it tricky to do so if your connection resets before you are able to complete the clone. There is currently no way to resume a git clone using git, but there is a neat trick you can use instead of cloning directly -- using git bundle files.

Here is how you would do it.

1. Start with "wget -c", which tells wget to continue interrupted downloads. If your connection resets, just rerun the same command while in the same directory, and it will pick up where it left off:

   wget -c https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/clone.bundle
   

2. Next, clone from the bundle:

   git clone clone.bundle linux
   

3. Now, point the origin to the live git repository and get the latest changes:

   cd linux
   git remote remove origin
   git remote add origin https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
   git pull origin master
   

Once this is done, you can delete the "clone.bundle" file, unless you think you will need to perform a fresh clone again in the future.

The "clone.bundle" files are generated weekly on Sunday, so they should contain most objects you need, even during kernel merge windows when there are lots of changes committed daily.

We are happy to announce that Fastly has offered their worldwide CDN network to provide fast download services for Linux kernel releases, which should improve download speeds for those of you located outside North America. We have modified the front page to offer CDN-powered download links, but all the existing URLs should continue to work.

If you would like to avoid using Fastly, you can simply change the URL to have "www.kernel.org" instead of "cdn.kernel.org". As always, please use PGP Signature Verification for all downloaded files regardless of where you got them.

Linus named the upcoming 4.0 release of the kernel "Hurr Durr I'ma Sheep" (see his git commit), so we are celebrating this April Fool's day with a minor prank. If you've been redirected to imasheep.hurrdurr.org, do not panic. It's all part of the joke.

We've also restored all FTP and Rsync access to the mirrors.kernel.org servers, as we seem to have resolved our SSD and dm_cache problems. If you're still using FTP, however, please consider switching to HTTP. FTP is a protocol designed for a different era -- these days everyone should be avoiding it for multiple reasons.

We've had to temporarily limit FTP access to mirrors.kernel.org due to high IO load.

We have recently upgraded our hardware in order to increase capacity -- 16TB was no longer nearly sufficient enough to host all the distro mirrors and archives. We chose larger but slower disks and offset the loss of performance by heavily utilizing SSD IO caching using dm-cache.

While it was performing very well, we have unfortunately run across an FS data corruption bug somewhere along this stack:

megaraid_sas + dm_cache + libvirt/virtio + xfs

We've temporarily removed dm-cache from the picture and switched to Varnish on top of SSD for http object caching. Unfortunately, as Varnish does not support FTP, we had to restrict FTP protocol to a limited number of concurrent sessions in order to reduce disk IO. If you are affected by this, simply switch to HTTP protocol that does not have such restrictions.

This is a temporary measure until we identify the dm-cache problem that was causing data corruption, at which point we will restore unrestricted FTP access.

Since we rely on the OpenSSL library for serving most of our websites, we, together with most of the rest of the open-source world, were vulnerable to the HeartBleed vulnerability. We have switched to the patched version of OpenSSL within hours of it becoming available, plus have performed the following steps to mitigate any sensitive information leaked via malicious SSL heartbeat requests:

• Replaced all SSL keys across all kernel.org sites.
• Expired all active sessions on Bugzilla, Patchwork, and Mediawiki sites, requiring everyone to re-login.
• Changed all passwords used for admin-level access to the above sites.

As kernel.org developers do not rely on SSL to access git repositories, there is no need to replace any SSH or PGP keys used for developer authentication.

If you have any questions or concerns, please email us at webmaster@kernel.org for more information.

If you would like to mirror all or a subset of kernel.org git repositories, please use a tool we wrote for this purpose, called grokmirror. Grokmirror is git-aware and will create a complete mirror of kernel.org repositories and keep them automatically updated with no further involvement on your part.

Grokmirror works by keeping track of repositories being updated by downloading and comparing the master manifest file. This file is only downloaded if it's newer on the server, and only the repositories that have changed will be updated via "git remote update".

You can read more about grokmirror by reading the README file.

git clone git://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git

[kernel.org]
site = git://git.kernel.org
manifest = http://git.kernel.org/manifest.js.gz
default_owner = Grokmirror User
#
# Where are we going to put the mirror on our disk?
toplevel = /var/lib/git/mirror
#
# Where do we store our own manifest? Usually in the toplevel.
mymanifest = /var/lib/git/mirror/manifest.js.gz
#
# Where do we put the logs?
log = /var/log/mirror/kernelorg.log
#
# Log level can be "info" or "debug"
loglevel = info
#
# To prevent multiple grok-pull instances from running at the same
# time, we first obtain an exclusive lock.
lock = /var/lock/mirror/kernelorg.lock
#
# Use shell-globbing to list the repositories you would like to mirror.
# If you want to mirror everything, just say "*". Separate multiple entries
# with newline plus tab. Examples:
#
# mirror everything:
#include = *
#
# mirror just the main kernel sources:
#include = /pub/scm/linux/kernel/git/torvalds/linux.git
#          /pub/scm/linux/kernel/git/stable/linux-stable.git
#          /pub/scm/linux/kernel/git/next/linux-next.git
#
# mirror just git:
#include = /pub/scm/git/*
include = *
#
# This is processed after the include. If you want to exclude some specific
# entries from an all-inclusive globbing above. E.g., to exclude all
# linux-2.4 git sources:
#exclude = */linux-2.4*
exclude =

# Run grok-pull every 5 minutes as "mirror" user
*/5 * * * * mirror /usr/bin/grok-pull -p -c /etc/grokmirror/repos.conf

Special thanks to Benoît Monin for donating a MIT-licensed CSS theme to the kernel.org project to replace the one we hastily put together. Though the Pelican authors have since obtained a free-license commitment from the copyright owners of the CSS files shipping with Pelican, we wanted to have something that looked a bit less like the default theme anyway.

If anyone else wants to participate, full sources of the kernel.org website are available from the git repository.

We've implemented two oft-requested features today:

1. The download links now default to .tar.xz versions of archives
2. There is now a JSON file with the release information located in https://www.kernel.org/releases.json. If you've been screen-scraping the front page, please use this instead.

If you have any other feature suggestions, please send them to webmaster@kernel.org.

Due to a failure in one of the rsync scripts during the maintenance window, the mirrors of /pub hierarchy on www.kernel.org got erased. We are resyncing them now from the master storage, but in the meantime you will probably get an occasional "Forbidden". The entirety of the archive should be rsync'ed in a few hours.

We apologize profusely for the problem and will fix the script to make sure this doesn't happen again.

Contents of git.kernel.org are unaffected.

You are probably wondering what happened to the site's look. Unfortunately, we've been alerted that the default theme shipped by Pelican (which we largely adapted) has an unclear license. Until this is cleared up, we've put together a quick-and-dirty cleanroom CSS reimplementation that preserves the functional aspects of the site, but sacrifices a lot of the bells and whistles.

If you are a CSS designer and would like to donate your own cleanroom style, please let us know at webmaster@kernel.org.

Our apologies, and we promise to keep a keener eye on licensing details of various templates distributed with open-source products.

Welcome to the reworked kernel.org website. We have switched to using Pelican in order to statically render our site content, which simplifies mirroring and distribution. You can view the sources used to build this website in its own git repository.

Additionally, we have switched from using gitweb-caching to using cgit for browsing git repositories. There are rewrite rules in place to forward old gitweb URLs to the pages serviced by cgit, so there shouldn't be any broken links, hopefully. If you notice that something that used to work with gitweb no longer works for you with cgit, please drop us a note at webmaster@kernel.org.