These are unedited transcripts and may contain errors.

---

The DNS Working Group resumed on the 8th October 2009 at 2 p.m.:

CHAIR: Good afternoon everyone. A very good afternoon Alex. For those of you who were not here this morning, my name is Peter Koch, I'll guide you through the second session of the DNS Working Group today. I am, as Jim already mentioned, one of the three co?chairs of the DNS Working Group, with the other ones probably being stuck in the Working Group chair discussions still.

So this is the second part of the agenda. We will first have a ten?minute overview on the RIPE NCC update to be given to us by Anand. Shane Kerr will have a presentation about his analysis of Lame delegations, actually for the reverse space, I guess, in the RIPE recently, as a follow?up to the action item that I mentioned this morning.

Then we will have two mentions: One given by Dwayne Wessels about the OR Groupzone zone... And after that, we will have an open discussion on all these root scaling issues, so we will probably take questions for K and L only for clarification, have a discussion afterwards. If there is time left we are going to have any other business.

So, that said, I would like to ask Anand to come up to the stage and do the presentation. We have a pretty packed agenda. And try to get through here.

AMAND BUDDHEV: Good afternoon ladies and gentlemen. I am Amand Buddhev, I will be providing the RIPE NCC DNS update.

For those of you who don't know us, just a quick introduction. We are a small team at the RIPE NCC. We consist of three people, /SOERD owe /STAOEUBG who is actually here; Wolfgang, he is currently on holiday so he is not here; and myself, I manage the DNS services department. So we have a number of services that we provide for, to the RIPE NCC region as well as the general Internet. We operate one of the 13 root servers, that's K?root. We also provide the reverse DNS for the DNS space we allocate. We do provide some secondary DNS services for some of the ccTLDs that don't have the financial or technical resources to, you know, run their own secondary DNS.

We are also responsible for the operations of the ENUM zone.

We also operate an action 112 node. More details of this coming up.

We have also been signing all our forward and reverse zones since 2005.

And we are also responsible for the management of the RIPE NCC internal DNS services, such as the ripe.net zone.

Some updates about K?root.

This year we have started cooperating with AfriNIC to expand K?root and deploying new instances in their region. We have already deployed one region and that's in ?? sorry one instance, and that's in /TKARey salam, and we have got plans to add new instances in map /AOUTey Mozambique, Kampala, others in the pipeline.

Other updates our instance in Frankfurt used to be what we called a local instance and we have now upgrade it had to the status of global, meaning that the prefix is available throughout the Internet.

We have also been busy with with software updates. We have done all kinds of OS updates and security updates. And we have recently moved to NSD version 3.

So, we are going to continue our plans for expansion. We are talking to LACNIC about possible cooperation with them to install some new instances in the LACNIC region.

We are also busy doing some preparations for the arrival of a signed Root Zone and a signed arpa zone. As he have heard from Matt Larson and Joe, this Root Zone is going to be signed quite soon. Arpa is going to be also signed by the 1st December, so we have got lots of preparations to do.

We are also busy with hardware replacements, old servers that need to be replaced. And we plan to announce the IPv6 prefix from K?root from even more instances.

A quick update about the Lameness project. At the last RIPE meeting, it was pointed out to us that we were sending e?mail alerts to some e?mail addresses which weren't quite responsible for some of these zones, we actually found a bug in our software and we fixed it. What our software now does is to look up the zone contact and the technical contact and send e?mails to them, and if not, it falls back to the MNT buy attribute.

We are continuing to run monthly checks and we publish these results to the RIPE NCC website and what we have done is we have lowered the alert frequency. So instead of sending monthly e?mails we send them every three months. At the moment, we are performing checks against over 1 million pairs. Zone and name server pairs and we find that about 5.5% of these are Lame.

There is another issue. We often get tickets from confused users. And the reason for this confusion is that there are over 15,000 domain objects in the RIPE database for the /24 boundary, which actually has /16 parents, and as you might know, DNS works in such a way that we cannot do delegation for the child if the parent is already delegated. So our provisioning system actually ignores all these and we get tickets from confused users. I have created a /24, why isn't my delegation visible?

As an example: This 192.94.in?A DD R.arpa. There exists a child and the provisioning system ignores them, causes confusion for the users. There is an action item with the database Working Group to actually disallow this and remove these objects.

This was an action item on the RIPE NCC. The RIPE NCC Trust Anchors are in the DLV, and they are still available in the ISD DLV. ISC believes that this helps the deployment of DNSSEC, and so they will continue to publish these in the DLV. We are satisfied that ISC makes every effort to check that they have imported the Trust Anchors correctly. But our recommendation is for users to continue to fetch Trust Anchors from the RIPE NCC website.

DNSSEC growth: It's been growing, but very slowly. Some numbers here from RIPE 54 to RIPE 59. We now have 189 signed zones in the reverse stream. But there is some Lameness. Out of the 181 zones, 38 have secure delegations, but there are no DNS keys in the zone. And two of them we found have secure delegations but this doesn't match the DNS keys in the zone. The result of this is that validating resolvers cannot resolve these reverse zones and what we intend to do is incorporate this kind of Lameness check into our Lameness project, and periodically alert users that their DS records may have expired.

We have plans to improve our DNSSEC infrastructure. We have a new signer, which we are going to be deploying quite soon. We also want to improve the colocation, make it more secure, and review all our operating procedures. When we do the transition from the old signer to the new signer, we will sign our zones using both of them. This will result in no changes to our policies and procedures and users will not notice anything.

We also want to improve our website. Change the documentation, make it simpler for users to fetch our Trust Anchors.

The ENUM zones operations are stable. There's been one new delegation since the last meeting and that's Jordan. Three of the zones are now signed and the Netherlands is soon going to introduce a secure delegation into the ENUM zone.

We operate an AS 112 node. This mops up all the reverse DNS queries for RFC 1918 space. We have installed new hardware and a new OS and this node is operated at M 6 and hits about 2,000 queries her second.

Thank you for listening and if you have any questions, please let me know.

CHAIR: We have a minute or two for some questions.

AUDIENCE SPEAKER: Denis Walker, RIPE NCC. You asked the database to remove /24s where there is an existing /16 and not allowed a creation, what happens if you create a /16 where there is an existing /24? Should we delete that one because it now becomes ineffective?

AMAND BUDDHEV: I think the main object would be rejected because that would also break DNS. So, it would work both ways. You wouldn't allow creation of the parent if the child exists, or allow creation of the child if the parent exists.

AUDIENCE SPEAKER: But there are ??

CHAIR: Denis, I am going to be rude, that's why I am here, I suggest we take this discussion off line and it should happen within the NCC in the first place, with community input of course. But this seems to touch upon a very seldom operational thing in transition. But thanks for bringing this up and it would be great to post?it to the mailing list.

AUDIENCE SPEAKER: I have two questions about the DLV interface.

One: Do you contact them when you do a key change and do you look at, do you confirm that they have made the change? That's one question.

AMAND BUDDHEV: We do not contact ISC when we roll our keys. We do not encourage ISC in any way to import our Trust Anchors into the DLV. Our position is that users should always import keys from our website.

AUDIENCE SPEAKER: The second question is: Do you know of any other Trust Anchor repositries, things of the DLV type that are out there that you are either aware of, concerned, that you might want to make sure they are up to date too?

AMAND BUDDHEV: I am aware of SecSpider which also fetc hes Trust Anchors and imports them. And I know this they have our Trust Anchors, but SecSpider is purely experimental and we don't see any reason to check if they have correctly imported and usage of SecSpider, if it causes any problems, it suesers caveat.

AUDIENCE SPEAKER: Shane Kerr. Just one quick question, otherwise I'll eat into my own presentation time. So, arpa is going to be signed, right? Is an IANA arpa?delegated from arpa or how does that work?

AMAND BUDDHEV: Yes.

AUDIENCE SPEAKER: Who runs it?

AMAND BUDDHEV: Another arpa is delegated to a set of name servers around the Internet which includes some of the regional registries such as the RIPE NCC.

AUDIENCE SPEAKER: I guess my question is: If arpa is signed and the RIPE NCC /8s are signed, but?? arpa isn't signed, then we haven't gained anything in the reverse space at least.

AUDIENCE SPEAKER: Is someone from ICANN ??

JOE: I can't speak about any current plans to sign in add /RA arpa now. Just as a point of information, it'S served by a subset of the root server.

AMAND BUDDHEV: Thank you for the correction.

CHAIR: It's probably not so much by which server it's served but who is the main tier of that zone in practice.

JOE: I think the top level zone is currently created by ARIN.

CHAIR: Thanks. So, you touched, I guess, two or three action items that we might want to close right now. The one is that about the Lameness issue, and I'd like to postpone that discussion until after Shane's presentation.

The second one that we just saw you had internal discussion with Denis about, any, are you in a position to keeping this open and running so that Anand can take care of this?

Okay. So this is kept alive and the last one is, to come up with a position and, or recommendation regarding the RIPE NCC maintained zones key material in ISC's DLV and I think we have seen a response here. You have tried to pursue this further with little success. So is there anything the Working Group would like to add to this or do you feel we can close this action item which would be my proposal? Everybody comfortable with closing it?

Okay. Thank you. Done.

AMAND BUDDHEV: Thank you everyone.

(Applause)

CHAIR: Thanks Amand.

And next on the agenda is Shane Kerr with his findings on a Lame delegation survey, and as I said, roughly related to a presentation of last time and a discussion we had to cut off.

SHANE KERR: So, I am Shane Kerr. I work at ISC and as Peter said, this came out a discussion from Lameness from last time, the last RIPE meeting.

So, the RIPE NCC implemented a system which checks for Lame DNS servers in the part of the reverse they maintain. This came out of an action that this Working Group actually requested this from a while ago, and I thought, well, it seemed like a lot of effort and it seemed like annoyance to people so I thought maybe we should check to see if there is a problem before we solve it. It's a bit sad, because we have actually already ?? the NCC spent a lot of time implementing the solution, but I said well, okay,. And then the RIPE NCC said, okay, we agree, why don't you go ahead and do that then? I said all right.

So, I didn't do this in a vacuum. The RIPE NCC DNS department, Amand and his people, provided data and explained how the system worked. Gave me access to the database and things like that. I actually used OARC DNS which is a research group for doing this kind of analysis of DNS data. It let's me do the analysis on a server without people having to worry if I am going to download it on to high lap tap and sell it on e?bay or something. Thank you to both RIPE NCC and the OARC for letting this happen.

What did I actually do? The RIPE NCC runs these Lameness checks and it's straightforwardment you do ?? because the RIPE NCC has the NS, the set of NS records for each delegation. They try to convert each of those into either and A or an AAAA and then they do a look?up for the domain at each of those addresses. So these are the kind of two places where things can go wrong and it's more sophisticated than that. They do it over a period of time and a number of retries and good things like this. Plus they have the additional work of having to figure out contacts. I don't care about that. For this activity, this is what's important.

So, what the RIPE NCC did was they did tomorrow packet captures at one of their primaries. And it was for a period of one hour dumps, I think it was one hour ?? yeah, over a day or so at different times so we could see if there was any trend throughout the day.

So, the actual methodology is, we have the actual, we have what people are querying and the answers we're giving and we have the known broken servers, what we want to do is see are people querying for broken servers or are they querying for servers that are good? The question is does Lameness matter? And it could be that you have a million Lame servers, but you only get one query a day, so why do you care? Or is could be that you have one Lame server but it gets half your queries, we are trying to figure out how this works.

How did I do this? Well, basically I parsed each reply. These are the list of name servers that were sending back in reply. This is a very small mismatch, because the Lameness checks run over a period of weeks, and the captures of a specific time. That ended up 3 NS in one hour of packet capture, it's a bit of churn but not a huge amount to affect the result. I am going to pretend that's not important.

If there is no NS this the answer we give back, then we are done, we're fine because there is no chance of Lameness affecting this query if there is no NS in there. NX domains, if there is no answer for that obviously you get that. The is, OA queries you get from your secondaries and things like that. And a lot of errors, not as many as I thought. This is all unaffected by Lameness. If there is an NS return, we check to see if Lameness affects it.

So, there is more than one way for a server to be Lame. Given that we had this kind of two step process, we can have an error in both steps. So the first is that we may not resolve to an A record. In the example that we have with up here, I just give some typos, which is probably a commoner roar I would assume. I don't know ?? I assume that there is not sophisticated checking when workers are added for this kind of thing, so this is the kind of thing we'll see. There is a million other reasons a record may not resolve. It could be just any of the servers are down. It could be all kinds of things. I did not go into the detailed analysis of why NS records didn't resolve to A or AAAA records. I categorised that as one type of error. If you did get an address, you may not be able to query at that server. That's kind of the second large category of errors. And you can see here this person probably mistyped their first object of their IP address.

So a little diagram which I don't usually do. Basically we start off. We get a query. We look at the answer that we get on the upper right there. It has a set of three NSs in it. Then we try to look at the, how we turn those name servers into the IP addresses. The bottom we don't have any IP addresses that resolve so we put XXX. Finally we look at the actual servers and we see that of the three example ones that I made up, two of them are responding okay and one of them doesn't respond properly. This is the kind of the process of looking at this stuff.

So again these two large classes of error, what is the impact on the the user if there is Lameness? If you have a bad NS, it could cause all kinds of things. For example, if someone mistyped their domain name like in one of the examples I gave, then you are going to get an NX domain right away and your server is going to reply requickly. You won't get an answer but in the case of reverse DNS, a fast negative response is probably okay.

Unfortunately you also could get time outs if servers are badly maintained or things like that. Or some name servers implementation, DJB, DNS, will just drop a query they don't understand which causes unpleasant behaviour on the resolvers, because they have to wait for a time?out.

Now, unfortunately we don't know ?? I don't really know, based on the data that we have, what exactly the impact on the user is going to be, so, it could be something like this. And so, the reason that I bring this up is because in my mind the worst thing that we can do is make the user wait 30 seconds for an answer. That's a real negative impact of Lame DNS.

So, we saw that we can breakdown our NS records, and because of the way DNS works, you have ?? in theory a good resolver will pick at random out of the name servers it gets back as an answer. So basically you have an equally distributed chance of hitting a bad name server, so if you have 2 NS and only one is bad you have a 50/50 chance of getting it bad. If there's three in one bad you have a little more chance. It gets a little more complicated if you have three name servers and two are bad. In the first query you have a two thirds chance of picking a bad one. So this is simple works you remember from your fresh man year at university.

Anyway, now, impact of bad server. This is when we did get an IP address and it doesn't answer properly. In my feeling is that most likely, in most cases, this is going to be because the server is not working properly and this is probably more likely to cause badness force the user. I don't know that that's true. This is just a guess. The further analysis will be required to confirm the suspicion, but my suspicion is that IP address that is not answering properly is probably a higher cost than an NS that's not working right. It's a delay for user. As far as Lameness goes, delay is probably the worst thing.

For bad servers, the exact same logic applies about your chance of hit ago bad server. So if you have a set of five IP addresses, then if one of them is bad, you have a one in five chance of hitting that at random. So...

I did this analysis for the reverse tree. I think a similar analysis could be done by anyone running a delegation only name server. If you care about Lameness of your other people that you are delegating for, the technique is pretty straightforward. If you want to try something like this, please talk to me, I can either give you the code or I can possibly run the analysis on your behalf if you are an OARC member right now. Anyway, that's just it. Please talk to me if you are interested this this kind of thing.

Getting back to the numbers. I didn't put commas in. 16 and a half million packets were scanned in this thing and of course as you would expect, half of them were the replies that we sent, the other half were the queries. There were a few unparagraphsable replies. DNS has all kinds of crap in it. The packet with non 0R code, these are mostly NX domains but they are also errors. The important thing to note is about non?0R code is Lameness doesn't matter because you are not getting a reply that affects the user. There were a bunch of packets with stale NS. These were the ones that I mentioned earlier were because we were running the capture at a different time than the Lameness check, I was able to detect that and kind of discount these. And then, we have a number of packets with NS in the RR set.

So my estimate. You can see the numbers there. And the reason these are not integer numbers is because of the problemistic nature of the replies I described earlier. So if one of your two name servers was Lame, I consider that about a .5 queries that's being affected. These are estimates and all further numbers that I am giving are estimates on what the impact on the user will be.

What does that actually mean? This is our chart, and the nice blue area there is my estimate of how many responses came back and the user got a successful reverse look up on the first try. I am not sure what colour that is, we'll call it wine, the wine coloured area there is, answers that are problem NX domains but they could be errors as well. Anything that's not just a valid response that says here is an answer you can use. These thin little lines there, are everything else. So the green one is, that's where you got an A or an AAAA record but you couldn't connect to the server and get an answer for whatever reason. And that little tiny yellow sliver is the NS look up failures, so the key message from this is that all of those errors are a tiny slightly sliver and don't really have a huge impact. So, about 0.3% of queries that users actually sent to the name server would have gotten a bad NS, would have been impacted by a bad NS. And 0.8% had an error because of these, the server is not actually working. So, my estimate, based on all this, is that about 0.1% of queries were affected by Lameness.

So, this is, I say estimate, because of course it's an estimate. The result do not account for caching. Caching could make things better or worse, we don't know what users are doing. My intuition tells me the picture from the user side is much better than the 1% because the result are going to discover the areas and cache the good results. But I don't know that. That's just a guess. I also have to admit that when I was about done with my result, I noticed that I was strictly looking at IP and not doing the full mapping of IP plus domain. So it may be ?? the results may be slightly better or worse if you have some servers that are answering properly for some demains and not others. So... yeah, if I run this again, I'll fix that. I don't expect that the results will be significantly different though.

As I discussd a little bit, the affects of the bad name servers. We don't really know ?? because of the data doesn't capture the full richness of possible problems, we don't really know what's going on with the bad name servers. I don't actually know what the effects on the user is going to be, which is the reason I left them as a separate field, rather than kind of collapsing this. The effects of bad name servers is slightly uncertain. We don't know what's going to happen from the user point of view.

So, what does this mean? As Amand said, about 5% of servers are Lame. Why did I come up with this 1% number? And I think if you think about what's going on in the DNS, it's pretty straightforward. If I have a server and it's not being used, then it doesn't matter whether it's Lame or not. I am just ?? it doesn't get fixed. I don't have time, and that's the reason it's broken. But if people are using their server, then people don't notice right away, but eventually they notice, and they do fix it. And so, the question is: Is 1% too much? I don't know. I tend to think it's a rounding error, the Internet is full of bad things. I did a brief check of DNSMON data for like a one day period and I found that the background error rate in DNS as recorded by DNSMON is about 0.3%. Now the average DNSMON rate was about 0.8 percent, which implies to me that in the DNS world, a 1% error is something we can live with.

Well, cost benefit analysis. So I have three proposals. The first is that we stop doing blanket e?mails. I think our ?? I think it's just spam, to be honest. Two POP ways to move forward are one: Is that we can produce a more targeted report. Most of this stuff goes through LIRs and we can perhaps ask the RIPE NCC to produce an annual report that they give each of their members when they send their bill, and say, here is a review of your DNS. By the way you are Lame here, if you want to ask us how to fix it, we'll help. There is already a form of communication that we can rely on there.

The final proposal is: It might be possible, I haven't done the analysis, it maybe may be that you have a certain number of servers which are causing most of the problem. Rather than send a blank e?mail you look at the query you are getting, find the ones that are causing the most problem and then don't send a blank e?mail but a specific e?mail to those users. Further analysis is required if you want to move forward on that and it is going to cost the NCC.

That's all.

CHAIR: Thanks Shane. Any questions? Comment?

AUDIENCE SPEAKER: I have a lot of comments about Lameness, I'll probably talk to you afterwards because I am included into the history of where this came from. But the question ?? I think on the other hand, your questioning of why do this actually is a valid question given what I know too. But I would disagree with how you got to that. This is a database clean up operation now. I think that's a way of looking at this. Is this worthwhile doing as opposed to trying to save the queries on the Internet? I'll leave that just as a general. What you are trying to do is make sure what's registered is correct. The name servers are correct and so on and there is a value to at that which may be totally distinct from saving packets on the Internet.

SHANE KERR: Why I do care? Why do I care if someone has bad data if it's not never accessed?

DANIEL: You may not care, I do. This is Daniel again. I think this community wants to project the image of having its house in order. And definitely we, at the RIPE NCC, we regularly get criticised by our antagonists for the crappey databases that we keep, which is totally unjustified actually if you care to other sorts of databases. So there is, I agree fully with what he'd has said, there is an intrinsic value in cleaning up the databases which we publish.

SHANE KERR: If the ??

AUDIENCE SPEAKER: Sorry, I made a pause to think. This is day 4 of the RIPE meeting. Bear with with me. There may not be an operational value, but there certainly is a policy and political value. Now I am finished.

SHANE KERR: If the RIPE NCC was bearing the cost of a clean up, then I would agree with you completely. However, the RIPE NCC is not bearing the cost. It's the DNS administrators who have to spend the time and resources to figure out what's going on and fix it.

DANIEL: This assumes that there is a difference in interest between those DNS administrators and the RIPE NCC. I think the RIPE NCC has no intrinsic interest in cleaning this up. I said very carefully, this community has an interest in cleaning up its databases and I think those DNS administrators, since this is reverse DNS, are quite clearly part of this community.

SHANE KERR: I guess we just disagree then.

CHAIR: Olaf, are you in line?

Olaf, more or less the same comment.

SHANE KERR: Olaf agrees with Daniel. Ah oh: Actually, I wanted to be a little bit more value neutral than agree with or be opposed to.

But, there is a point about as an industry, making sure we have our act together if it comes to issues of consistent management of everything we do. And that is what I heard Daniel say. Now, as a community, you might agree or disagree that RIPE NCC has a role in it. This is where I haven't made my mind up myself. But I think that the point that we as an industry have a joint responsibility to basically keep our act together in multiple aspects of running the Internet. You know, it's an important message. And that's what I agree with.

SHANE KERR: Okay. I guess, I know I am over time and I apologise, but I guess I'd just like to end and say I understand the desire to want to keep your house in order. I think what we fail to do was think about the cost benefit, and I consider the benefit very low. If we look ?? assuming my estimates are right, and I think that because of the effects of caching, we are looking at significantly less than 1% of end users who are actually affected, and I think if we compare that to the background error rate of DNS, it's about the same. I think there are better ways to spend our time and effort than looking at Lameness. That's all.

CHAIR: Okay. Thanks Shane. I think you got that message through. Nonetheless, we saw that in Amand's presentation, I guess much of the criticism/suggestions that you made last time had been taken up, so the error was corrected. The selection of targets for the e?mail was tightly focused not to involve too many parties that couldn't respond, and the frequency has significantly been reduced. You have the three proposals on there. I think the annual report to the LIRs might be an interesting thing. Actually I am inclined to try to throw that over the fence to the NCC services Working Group, but I guess we need some more discussion on this. The blanket e?mails versus the suggestion by Amand. We need to combine these a bit here. My proposal would be to first of all that the RIPE NCC be, given the chance to get some experience with a null implemented schedule, but at the same time, and that's happening today I guess, be very careful in seeing these signs of harassment or embarrassment that you mentioned like people guess these e?mails and they have to actively ignore them. Is that a path forward? What does the Working Group think? I am just not happy to do voting in either way here because we just don't have enough ??

SHANE KERR: I am quite happy to do this on the mailing list as well.

CHAIR: Okay. So for that matter, can we then actually close the action item that was on the NCC and you for doing the analysis and coming back and fixing errors, is there any objection to that?
No, okay. Anybody still awake by the way?
Thank you.

So we close that action item so far for the minutes. And I think we'll take that to the list and start out from there, with some experience gathered and feedback from affected operators.

Thanks, Shane.

(Applause)

Next on the agenda is Duane Wessels from OARC who will present us findings. I love this new vocabulary, from the Root Zone impact analysis that was carried out by OARC under contract with ICANN.

I want to reiterate that Geoff did the bulk of this work. I am just lucky enough to be able to be here to present it to you today. We say we say and by the way I expect to be skipping some of these slides.

As has been said, there are interesting times ahead for the DNS root. Lots of new and exciting things coming up with additional IPvs exclude, DNSSEC, IDNs and more TLDs as well as continued growth and continued Anycast expansion.

So, earlier this year ICANN contracted with OARC to do a study which similar lates changes to the Root Zone looking at five particular areas. First, just simply the size of the zone file itself. Or the zone data. 2. How large a zone affects server response latency. How it may affect server start and restart times. Bandwidth requirements for zone transfers and how these changes might affect truncation and increase TCP.

I'll probably talk about 1, 4 and 5 and I may skip over 2 and 3.

Just a note about the hardware that we used. Oh arc has this test bed with 16 big servers, each servers has 4 course and each has 16 gigabytes of memory, although for one of the servers we did upgrade it to 32 so that we could test some of the larger zones and we have some gigabit switches.

The software that we tested is bind 9.6, as well as NSD 3.2.1. The operating system was usually Linux sent os, we did a couple of things on FreeBSD. They are tools that we used were DNS perv, TCP replay, NISTnet and some custom scripts.

We tested quite a variety of zone configurations. There were five different types of zone content. First is an unsigned zone sort of the current state of the zone with mostly IPv4 glue. The second is also unsigned, but with as much IPv6 glue as there is IPv4 glue. So, in other words, every NS circuit has both types of addresses. Then we have a signed zone, with both types of glue. And where 10% of the delegations have DS records. 50% have DS records and 100% have DS records. For each of these five types of content, we tested five ?? well, we had five different sizes of zones. 1,000, 10,000, 100,000, 1 million and 10 million. We didn't exhaustively test all of these combinations, but we did get pretty good coverage.

So the first test was to look at memory usage. And the way we approached this was to look at how big does the name server process get in memory when you load these zones? And on Linux we used this utility called P map which uses the memory used and shared libraries and other sorts of things.

Here is a graph that shows the memory usage for bind for all of the types of zones and all of the sizes of zones and note that both axes here are logrhythmics. But you can see, as would you expect, more or less, proportional ?? the memory use is proportional to the zone size.

Here is the same for NSD. And, if you cycle back and forth, you see that they are more or less the same. One thing you might note here is that we don't have any data for the signed 10 million TLD cases, because we didn't have enough memory to load it on our largest server. So it took more than ?? it would take more than 32 gigabytes of memory to load a 10 million TLD signed zone with NSD.

The conclusions here are the memory usage, or the zone data is proportional to the number of TLDs. If you have a fully signed, fully glued zone, use about twice as much memory as an unsigned zone of today. And as I mentioned, this little wrinkle with NSD.

Next thing we looked at was response latency. So to tackle this we synthesized some P cap files with queries whose characteristics were derived from some diddle data taken earlier this year, we replayed those queries with TCP replay and in all cases the queries were replayed add a constant rate of 5 thousand per second. This graph shows the behaviour of bind with an unsigned zone of five different sizes. Note that here the X axis is linear, and if you can read that, it's in milliseconds. So this is latency, sort of seen right next to the server, the time taken between a packet you know going into the box and then coming back out. Also note that it's quite uniform between all the different zone sizes, there is not much change there as the zone gets bigger.

Here is the same for NSD. An unsigned zone, or unsigned zones of increasing sizes. There is a little bit of variation there as the zones get larger. You know, the histogramme shifts down into the right a little bit. But again note the X asix is linear and the scale is a little bit different than for bind. So slightly better there.

This graph shows bind's behaviour for a signed zone, and here you can see that there is actually some significant change when we get up to like 100 KTLD zone, the performance starts to drop. It's a little bit hard to see on this graph so I also included a cumulative distribution function. The blue line in the middle is sort of the interesting case here. So if you also note that here the X asix is now logrhythmic, still in milliseconds. But when you get to, you know, we have here say 1,000 milliseconds or one second. In this case, you know, something like 30% of the queries were not being returned or timing out.

And here is the NSD behaviour for the signed zones. Not quite the same problems. A little bit of degradation in performance but not quite to the same extent.

And here is the cumulative distribution for NSD.

Also, note here there is, at the end there, there is a 4 and a half million TLD zone. As I said before, we couldn't get a 10 million TLD zone loaded but we were able to test this with a 4 and a half million TLD zone just to have something to put there.

So, for this task, bind's performance is stable for all sizes of unsigned zones, as is NSDs. Bind's performance degrades a little bit with the larger signed zones. And this is a problem that ISC has already identified and working on a solution, I don't know if Shane can speak to that or not, but?? maybe later. You can sort of tell us status of that if you like.

And just to sort of address this a little bit more. I want to emphasise that this performance issue is only a problem with NSEC zones, there is no issue with with NSEC 3. This is only an issue with a zone like the root where it's likely to have a large number of glue owner names in between non glue names. And it only becomes a problem when the zone gets you know something of the size of 1100 thousand TLDs. So this is not anything that we need to worry about today. You don't have to panic and start upgrading or anything like that. I think there is plenty of time until this fix will really be necessary.

Here is a cut and paste of one of our zone files that we used in the test. Just to show, just to highlight the problem. So here the come delegations, this is the authoritative data. And because we ended up using NS records that end with dotcom. Something like 10,000 of them get inserted between there and the next TLD that inserted there.

I am going to skip over this a little bit and skip to the conclusions of task 3. There is not a lot of surprises here. The start and reload times are proportional to the zone size. One thing we found was although bind could load the 10 million TLD zone within 32 gigabytes, it could not reload it within that much

I am also going to skip over this a little bit, because they tend to be a little bit boring. You know, everything is proportional to the zone size. One interesting thing we did discover, is that when NSD is used as the master it uses name compression, so that results in in bandwidth savings. We have some data there about what happens to zone transfer times in the event of packet loss and so on.

So, last I'll finish with talking about TCP stuff. We took some actual client traces from the DITL data and replayed those against servers with different types of zones. The first graph I want to show you is so that here, I think the client trace actually came from L?root. This is surveying a signed zone. There is a note there about abnormal queries being removed and that's because in the data we collected, there was a small number of clients that had some sort of aggressive behaviour that were skewing the results. So those were eliminated.

So the point of this is that if you took ?? this is with sort of a today's zone, if you took today's zone and signed it, you would expect to see something like 0.3% of queries being truncated and probably being retried via TCP. Most of these turn out to be NX domain responses, a small number were queries for the root zone with query type of "any."

AUDIENCE SPEAKER: Daniel. Short question. Definition of truncated. T bit set or additional information omitted?

SPEAKER: TC bit set.

So, that was with a current size zone and this graph shows with, both with a current size zone and with a 1 million TLD zone with lots of glue and lots of DS records. And in this case the amount of truncation jumps from 0.3 to 0.7, 0.8, something like that.

This shows a histogramme and cumulative distribution of sizes of responses that would have been truncated. So to get this, we took the client trace and picked out all of the queries where the DL bit was set and the EDNS size was set to 5. Then we changed the EDNS size to the maximum so they wouldn't get truncated and we measured the responses of those packets. There is a spike somewhere around 650. And now off the top of my head, I forget exactly what that spike is from. It's in the report. Then there is another spike closer to 700.

So, based on this, we sort of concluded that a Root Zone could expect something in the order of magnitude increase in TCP when the root becomes signed. For A root we measured, going from five queries per second to more like 50 per second. We noticed that a larger zone caused also an increase in truncation tore TCP traffic and we think that's due to longer names in the NSEC records because we assume that went you get to something of that scale with a million TLDs, you are going to have longer TLD names. And then based on the last graph, the UDP response that is might be truncated would be no smaller than say 125 bites if they weren't being truncated.

So that's what I have to present today. There is an URL here. If you'd like to read the full report or e?mail addresses for us to get in contact with us.

CHAIR: Thanks Duane very much. For the moment clarifying question. Then another presentation afterwards. Okay. No questions? Thank you very much Duane.

(Applause)

CHAIR: And I'd like to ask Lyman Chapin to come on the stage to give a presentation about the Root Zone study findings. Another effort contracted by ICANN.

LYMAN CHAPIN: I know we are running a little short of time. I want to be sure we have plenty of opportunity for discussion. So in part because I presented a lot of this material on Tuesday during a Plenary Session, I'll be as brave with this as I can.

A recap. What I am going to focus on here today is some of the implications of the findings that I talked about on Tuesday. The scaling the root report is available, has been since September 7th, and it's a product of a study that was undertake he be by the six people that you see on this list. Most of those six people are still here. On Tuesday I said that they are all here. Most of them are still here. Just a quick recap.

What we were talking about when we talk about scaling is a combination of four factors, and of the four factors that are on here, for the discussion that I think we are going to have today, it's worth pointing out that there are two of these are in some sense, happening already. We heard Matt Larson and Joe Abley tell us on Tuesday that the Root Zone will be signed. Full stop. And we also can probably conclude from the circumstances surrounding the exhaustion of the free pool of IPv4 addresses, that it's unlikely that we are going to see a scenario in which, for example, we start refusing requests from TLD registry operators to add AAAAs for their name servers. That just seems highly unlikely. So, of the four things that we're dealing with, it looks as though DNSSEC and the new address records and glue for v6 are pretty much foregone conclusions. Keep that in mind when we talk about what options we might have for dealing with the consequences of the study we undertook.

I won't go into the detail on the model. We can come back and refer to this if necessary when we have some discussion, but essentially we have a model that involved, that incorporates all four of the domains in which the root system plays out and the six principal actors.

Run through the findings again.

The most important of which of course is that it is extraordinarily difficult to perform a quantitative modelling simulation of the root for the simple reason that the root is both dynamic and decentralised, so the way in which with the individual pieces respond to changes from a combinatorial standpoint quickly becomes intractable from a quantitative modelling standpoint. We can do some quantitative models and some very useful studies, and Duane has described one of them. It's not the quantitative models and numbers are not useful. It's they are not useful in helping us make long term decisions in how reought to be looking at the evolution of the root system as a whole. It may be useful as a tool to help us understand in particular the near term effects of taking certain actions, but it turns out to be, unfortunately, remarkably unhelpful in drawing conclusions that apply in any meaningful sense so the root system after it has evolved past to what amounts to a very short?term. And therefore, we are proposing that the goal of oversight of the root system, the responsible goal of oversight should be instead to develop this concept that we are calling an early warning system, which involves both deciding what markers of stress are both genuinely meaningful, in the sense that they give us some actual useful intelligence about what's happening, and are measurable. And many of the metrics are probably available today but we haven't gone through the exercise of determining for each of the actors in the root system what are the things that would be useful to keep track of and how would we go about measuring those and coordinating the measurements so we would have a complete picture.

There is a second question that goes along with that which is, having arranged to take those measurements and having arranged to collect them and, you know, have somebody look at them, what sorts of actions might the community be prepared to take if, for instance, one of the metrics were to cause an alarm condition or otherwise exceed a threshold that we decided was significant? Those are two separate questions, I suspect the second of those two is going to be harder to answer than the first.

We came up with data to support two conclusions that I don't think will surprise anyone. The first is that on the provisioning side, which is the process whereby new information changes and new delegations and so forth that arrive as requests at IANA find their way through the Root Zone management system into the definitive Root Zone database that's maintained by VeriSign as the Root Zone editor. And that process is completely dominated by the human intervention steps, by the requirement that exists in and which will continue to exist for the foreseeable future as a matter of policy for there to be a human inspection of every change that takes place. And that human inspection takes place at the moment and again will continue to take place for the foreseeable future absent a change in policy in IANA, NTIA and VeriSign so, in all three places there is a human inspection of every change that goes into the root.

On the publication side, not surprisingly, the effect of increasing the size and/or volatility of the root is felt primarily at the end of what we have, you know, called long thin pipes. So, locations that are either poorly connected in the sense of either low bandwidth or very higher roar rates, or are poorly provisioned in the sense that it's difficult to put, you know, state?of?the?art servers and over kinds of equipment at those sites.

And the net of all of the investigations that we did, is that it's relatively easy to demonstrate that all of the actors in the Root Zone system would be capable of absorbing an increase, and by increase I mean both increase in size and volatility, on the order of hundreds. And that's a deliberately vague term. We are not talking order 100 in the sort of algorithm complexity sense, 100 times. We are talking about an increase of hundreds of new entries and/or hundreds of new change requests on an annual basis. That level of scaling could be absorbed by the system without any of the actors having to go into what we call a replanning phase where they would have to decide to do things differently, either from a personnel standpoint or from an equipment and infrastructure standpoint.

But when you get to order thousands, in other words, thousands of new entries or thousands of new change requests, the message is not that you can't do that, in other words, that something will break if you do that. It's that one or more of the actors in the system would have to do something differently, and of course, that's something that, you know, one or more actors could decide to do given sufficient will and resource and funds and so forth. But it means that the normal course of events, the normal way in which organisations typically do refresh equipment, or new hires and so forth, would not sustain that level of growth. So one or more of those organisations would have to do something differently.

And this is the slide that I put up on Tuesday to describe this concept of a replanning region. We worked with a model of dynamic operating regions that essentially identifies a range of scaling within which an organisation can deal with with what we call sort of run of the mill, or normal everyday increases.

And then there is a discontinuity at which point an organisation would have to replan to accommodate either larger ?? well larger load. And that replanning might involve deciding to double the size of the organisation by hiring lots of new people, or to, you know, buy and install and provision a different class of hardware, but they are the kinds of decisions that an organisation can't make without going through some sort of fairly substantial process of replanning that might involve you know some higher level manager signing off on things and so forth.

There is a third region we hope we never get into which is one in business the basic architecture of the system as it's currently constructed would not be capable of handling the load even if each of the actors were willing to spend lots of money and do upgrades. That's a situation in which, for example, it might become impossible to operate the provisioning system with the level of checks and balances that currently exist. And then you'd be in a completely unchartered area where you'd be trying to determine how to deal with with with with the desire to a near zero error rate in the provisioning process through some other means. A means that is in place right now works very well.

Just a couple of the effects of Root Zone changes. Because we looked at, again, the effect not just of adding new TLDs to the root but also adding DNSSEC and IDNs and v6 glue, and I won't go into these in detail. These charts are also in the report. But, the interesting thing about this and this second chart right here which shows what the impacts are, is that the combination of effects is more significant than any of the effects individually. But they are not as, sort of, negatively sinner gist I can, if you are, as we thought they might be. For the first part the effects that are felt I adding new record types for instance to support DNSSEC are precisely what you would expect and they are not really exacerbated or otherwise enhanced by any of the other factors. So it is possible to talk about each of them roughly in isolation. There are some combined effects but they tend to be relatively small. And again we can talk about those in detail during the discussion. I am going to continue to go through this pretty quickly.

I'll close with the dedication page that was in the report in the version just prior to the one that we released. Just so that, you know, sort of an antedote, taking ourselves too seriously here, because we talk about these things. They are all ?? some of the things we have been talking about in the course of the study sound pretty grim and I think we need to keep a little bit of perspective. We are all doing a pretty good job and we are going to continue to do a pretty good job and we'll be able to fix this.

I'll stop there.

CHAIR: Thank you, Lyman, for that presentation.

First of all, I'd like to have specific clarifying question to this presentation. And absent these, I'd like to open the floor for general questions and discussion on this topic, and I am trying to be a bit organised regarding queue management. I'd like to ask people to line up at the microphone and then indicate by raising their hand that they are introducing a new topic, or a new issue to they can keep the discussion focused and ?? well...

First of all, clarifying questions to this presentation? None? Okay. So then, thank you Lyman again.

(Applause)

CHAIR: You are not relieved. Then the floor is open. Please line up at the mikes.

AUDIENCE SPEAKER: Jim Reid. Random member of the public. I think the root scaling work that's done is a great piece of work, I am a bit concerned some of the information might be used out of context. However, let's put that concern to one side. The key points that I took out of the survey was that the scaling group is saying that ICANN is essentially got a choice to make. Over the next couple of years, we can have a signed root, we can may be have IDN CCTLDs, we can maybe have more IPv6 at the root or new GTLDs, but we can't have all four of those things so I think that's needed to could is tackle those things in a priority order and I think it's also ?? I think these things need to be prioritised and obviously there is going to be competing demands on people's priorities.

But one of the most important things that needs to be clarified in my opinion is that these tasks are done one at a time. It's bad operational practice to try and change two or more variables at the same time. The combination are quite tricky. There could be unforeseen second order effects. I think the basic engineering principles we have to underline is make one change. Measure its effects. Satisfy ourselves that everything has reconverged back to a stable state for some definition of stable and then proceed to the next one and let's get through these things until all these things are in the root change plate are then dealt with. That's my personal opinion. And I wonder what other folk feel about that.

CHAIR: Any responses to that?

AUDIENCE SPEAKER: John. I'll respond to Jim's point. In a previous discussion of the root scaling study it was pointed out to me that 20% of the TLDs in the root already have AAAA records. That AAAA records are added to the root on demand and that's the way it should be and no one has advocated stopping AAAA records to the root. I think a strict interpretation of do only one thing at a time doesn't fit with the ongoing process of adding AAAA records.

AUDIENCE SPEAKER: Olaf. Strictly one thing is very difficult to define in an evolutionary environment such as, which I think the root is. Things are added fairly slowly now. New TLDs are added. New ccTLDs are added because new countries start to exist. IPv6 glue is added. DNSSEC, that is a real change. That is a measurable step function. There are other things which will increase the pressure on the system which increases the flux of changes, the scale of changes. Those might interact with with each other, and I am not quite sure how you would say first do these and then wait until you figure out what the impact is. So, the wording you phrased seemed to me to be a little bit imprecise.

AUDIENCE SPEAKER: Jim Reid. You are right, it may be a little bit imprecise. Perhaps that's because of maybe not completely thought everything through to the extent that perhaps I should have done.

I do think, however, you are right in a sense to say that added more IPv6 could be considered for a continuous process and adding a new gTLD from time to time could also be just a continuous process. But I think we are going to see a significant step change there, rather than just say, for example, one new TLD every six months of the year, so or additional AAAA glue record going in there, we could see large numbers of these things being embedded in a very short space of time. That I think becomes more of a step change type of thing rather than a small amount of continuous type enhancement activity. That's what I am getting at here. The thing I like particularly about IPv6 is we are going to have a fairly panic fairly soon because the world is running out of IPv4 addresses and we may have to be in a situation where lots of IPv6 gets added to the root in very short order so that everybody is ready for the IPv4 crunch.

AUDIENCE SPEAKER: Rob Blokziji. I think of the four changes presented and studied in this report, they are of two different classes: Signing the root we know exactly how big that is, the increase in the Root Zone. It is not unpredictable. Putting IPv6 glue in the Root Zone, in principle, you know exactly how much extra information that carries. Having 30 or 40 IDNs added, that is about the level of requests from the community, again it's quite easy to estimate how much change in the size of the Root Zone and how much burden on all the processes will involve. Of course, adding new G TLDs, is an enormous question mark because nobody knows how we talk about 100, 1,000, 100 million. And there is an interesting thing, which is an non?technical issue, but in the ICANN environment, the US legal environment, once you open that window, it is very difficult, I think, to close it. So that is, for me personally, the most interesting aspect of this whole thing to see evolving, and I think it's also the most dangerous one of the four.

The other three we can, I am convinced we can do in a controlled fashion, not all through on the same day, but in a well?controlled defined manner, but the real question mark and the real challenge to be frank, is how to handle a flood of 10 million G TLD requests. It might be in the report also mentioned this. That we might be saved by the fact that we have silly procedures which have a lot of human intervention per request. That will maybe trickling down.

CHAIR: Okay, thank you. So we are still on the topic of the relative impact of the four suggested additions.

AUDIENCE SPEAKER: What I got from the two presentations is that ?? Alexander Mayrhofer from the Austrian registry.

What Duane said that essentially there are not any technical limits in let's say scaling up the root to the equivalent of 10 GTLD. That's what I got away from it. I agree with Olaf that the step between an unsigned and a signed root is probably the scarest one because it's going to introduce a whole bunch of TCP queries, that's I think the greatest impact, and it's going to increase the size of course, and I don't expect for the new GTLDs because of the very complex process because of the huge effort of about 1 million euros involved in getting such a new TLD, if you take ?? yeah, well, if you include all the lobbying and all the preparation of the documents that you have to do ?? anyway... so I don't expect there will be millions of new TLDs in the first round. So, if I would have asked how many domains would be the maximum to expect, I would say it's a couple of 100 at maximum.

CHAIR: Okay. I have a clarifying question now to, maybe, Lyman and his team because you are touching upon an interesting issue.

AUDIENCE SPEAKER: Sorry one short point Peter. I fully agree that the administrative problems, how the Root Zone is managed are a big issue, because ?? and even ICANN and IANA going to review every signature once you have a newly signed record?

AUDIENCE SPEAKER: The answer is no.

CHAIR: The clarifying question I have is: Was or was not the assessment of probabilities of the shape of the curve part of the study, shape of the curve meaning what you said, where will it end? How many GTLDs will be there? And I think to that extent, that should be clarified.

LYMAN CHAPIN: We very specifically, and this was because we were told to do it this way, we very specifically did not rely on any contractual or administrative sources of friction in the system. So, for instance, people have observed that it takes ICANN, historically, between 6 and 12 months to conclude the contract negotiations with a new registry operator. If you imagine that introducing new TLDs was, you know, could not happen any faster than would be allowed by a process that completed contract nexts, and you know let's take the optimistic side and say six months. You would have a different scenario if you were looking at a system from a technical standpoint can sustain a much larger number of TLDs. No on the one hand you realise that the technical parts of the system, the software and the hardware and most of the people processes, particularly on the distribution side, the query side, can handle very large zones. Obviously we have very large zones that are being served for other, for some of the TLDs already. If you have that observation on the one hand and on the other hand, somebody, you know, brings up the point that the administrative process that ICANN uses right now to actually introduce new registries into the mix of registry operators takes six months to conclude a contract, you have got a big disconnect. And our study deliberately, again, under instructions, deliberately did not consider the administrative friction in the system. We considered only the friction that might be introduced within Root Zone management process by the way in which each of the actors conducts its business. So we took into account the fact that IANA has, you know, manual checks and it has personal relationships with administrative and technical contracts. We took all of that into account. We did not take into account the contracting and other sort of surrounding administrative issues that might also introduce delays.

CHAIR: Thanks very much. Same topic?

AUDIENCE SPEAKER: Same topic, in reference to Jim's original statement about isolation of variables as you make changes to the zone file. I think the principle is still sound except it has to happen in a new context, where you have changes that may be happening small and slowly and you have a sense of how those work already. If you are going to inject a big change like signing the root, that has to be done separately, as opposed to signing the root and a big ramp up of something else. So I think Jim's principle is still sound, it just has to be done in a different context because changes are happening all the time.

CHAIR: Thanks. Daniel?

DANIEL: I have one to this topic, because I think that there is one thing that worries me as well, Alexander said I take away from all this you can do 10 K. That is not what I read in that study. So, that's ?? just to make that clear.

Also I'd like to say that I have great admiration for the team that did that study and for the level of insight and for the level of presentation that they did in the short time they had. I think they deserve recognition for that.

(Applause)

So now for the new topic I wanted to, if I am the next one ??
This has been bothering me for sometime. The technical study that we heard suggests that the level of TC bit set and the level of queries arriving at the root service with TCP transport is not expected to rise significantly. Now, we always want head room, and we always want to plan for sort of discontinuiities as far as root name servers are concerned.

Let's take the hypothetical case that as some root name server, the level of TCP queries that arise and the amount of stake that needs to be kept to deal with them, affects the service level on the UDP side. What is the opinion of the room, or of the community, what should the root name server do? Should they stand by and take their hands off or should they go and prefer queries arising over UDP and EDNS and all that kind of stuff? That was the question number one.

Question number two: Would it be useful for Root Zone name server operators to make statements about what they intend to do in those cases before it happens?

OLAF: A clarifying question, Daniel. If you say service level on the UDP side do0 you mean you just have to wait a few milliseconds longer before you get a response? Or do you mean UDP, there is a high fraction of UDP packets that do not get an answer?

DANIEL: I think segradation of service. It's either they get dropped on the floor or they get serious delays. If it's not significant, don't do anything. But in my worst case scenario, when I look at how implementations work, and you are close to one of them, then it might actually lead not even through the name server software, what happens in the kernel and all that kind of stuff, it might lead to packet drops, incoming or outgoing. That's the thing I am worried about.

OLAF: So to inject something that ?? I haven't made up my mind about this, but some of the arguments that I would take into account, which I do take into account while making my mind up is actually: Is TCP traffic the cause of TCP traffic ?? is the cause for that TCP product ?? I have to start again.

Is the cause for TCP traffic a problem that is caused by a difficult word ?? I don't know, noncompliant things at the edge, or in between?

DANIEL: Very likely.

OLAF: How bad do you want to treat that ?? it's a difficult word, because we don't actually measure compliant or not, but how would you treat non?compliance in a general case?

CHAIR: I think you are deviating from Daniel's question a bit. Anybody responding? I'll let you judge that, but...

SHANE KERR: There has been a continual fear of TCP. The only hard studies that I have seen have kind of been pretty ad hoc testing people that done on individual servers. Having said that it looks like modern systems are quite efficient add TCP, you are looking at maybe you are able to run a third as many query using TCP as with UDP. That happen sounds pretty bad until you consider that 90 plus percent of the stuff that you are getting on your Root Zones is just gorp. So I don't suspect that when you get bow does queries you are going to have to fall back to TCP in that case.

DANIEL: My response to Olaf is, as a root name server operator I put that hat on very firmly now, I think it's already 10 yards down a slippery slope to make any categorisation of queries in, due to any criteria just as conform aunt or non conform aunt. I don't think any root operator, root name server operator would want to go there.

OLAF: Cannot tell from the packets.

DANIEL: Even if we could.

AUDIENCE SPEAKER: I am happy I can at least each up to the microphone. Patrick Felstrom.

I was part of the helping writing this report, thank you very much, Daniel, and others for the ones that liked the text, although it might be confusing sometimes.

I just want to point out, add something to the discussion on what happens with this TCP responses and whatever, that when we were discussing with the people the problem with what happens if it is the case that the respond size increases, that people, even in the technical DNS community have very different view of what actually happens. Is it fragmentation of the UDP packets that is the problem? Is it the increase of the TCP queries that is the problem? Etc., etc. Etc.. it's the TC bit in the packet or a combination thereof? And this is just, and just because we sort of disagree what actually will happen with different kinds of things, it's so problematic to also have the next discussion which we are trying to have now, which is what are we going to do about it. And this is one of the main reasons why we felt that the main finding which is so important is that we can do a pretty clear ?? we can see pretty clear what's happening with the data we have today, a very short period in the future but it's a very hard to pick because there are so many variables.

DANIEL: You may note that I quite carefully phrased my question, for the hypothetical case that the TCP loading would seriously affect the UDP loading. I was not going anywhere whether it was likely to happen or not. The thing I am concerned about as a root server operator, heaven forbid will happen, that one half of the community will cry at us and say drop all those TCP queries because UDP is the only real ones, right, and ?? that may happen ?? and on the other side, if we do that, then lots of people say hey I had an implied contract to serve all those TCP queries, what I am trying to get is a feeling from the community, like, in that hypothetical case, what do you want us to do? I don't want the answer now. I just wanted to raise this question.

Absolutely, it was not my intention to disagree, I was ex complaining why the discussion was so hard. Let me pick one of the things we studied in our report. What will the impact be when the size of the primary query gets above 512 bites? I claim that we should not have the discussion here, but I claim that we have at least five different responses in this room and that doesn't make the extrapolation easier.

CHAIR: Let's declare the priming size out of scope. Otherwise ?? I would have to run to the microphone otherwise. Next is ?? this is your last chance to go to the microphones, because I'll close the quay in 30 seconds. We are running into the break anyway, and we will have one item and under any other business and then I see Rob opening a new topic and Jim not being sure opening a new topic and I guess you were first.

AUDIENCE SPEAKER: Jim Reid. The other topic I would like to open up, to go back to one of Daniel 's earlier question is how should we publish things about that? I am not a root server operator, I wouldn't tell them how to run their servers, but I do think it would be unwise to tell people to how to mount DDoS attacks again the servers to make it more successful. I'll pass the mike over and I'll let other folks speak.

CHAIR: I think Bill was the next one.

AUDIENCE SPEAKER: Bill. On Daniel's issue, because the performance tuning requirements of TCP and UDP are relatively different, what we have been working on is splitting the TCP and UDP traffic from each other at the routing levels so that he can direct them to such server processes running under different kernels, but we haven't gotten too far with with that. We haven't put anything into production yet. So I am curious whether anybody else is doing the same thing. We'll try and write it up when we have something further.

AUDIENCE SPEAKER: Amand. First, DNS goes over TCP and UDP equally. So, you can't just turn one?off in the benefit of the other. That's not how it works.

Second, I have seen ?? I have paid close attention to the evolution of statistics of two signed zones and yes, the delay to DNSSEC was turned on these zones both saw an increase of TCP queries to about 10 percent of the total query load. Either of these zones gets more queries per second than the Root Zone overall. None of these zones had ?? an interesting topic was that since that day both have shown a decrease on the rate of query, so I don't know why. I'll probably never know why. But the worst day is the first day and in none of these situations has there been a problem, so I think we are freaking out about something that's very much in the hypothetical field.

CHAIR: Shane, same topic? Then Rob was first.

ROB: Change of topic.

OLAF: Essentially what Joao just said is very important, and in fact, I would like to see good data and real measurements about the impact and some analysis on the causes why these jumps in TCP traffic have happened. To be honest, I haven't seen them.

ROB: Change of topic and field we have been discussing. Some very interesting technical details of DNS provisioning. I now step over, on to another planet and that's called politics. We have, I think, an excellent report coming out of the technical community supported by the technical community, at least as represented today here. This report, one day will be formally submitted to ICANN, to the ICANN board, and they are there to take decisions. I want to express my personal worry, because I am not completely one hundred percent sure, based on past experience, either as a board member or observing the board for many years, that the right positions will be taken, because there are other pressures than those from the technical community being applied to the ICANN board in this process of reaching a conclusion.

So, I think we should, as a community, can do two things: In the first place, make it be known that we fully support this, the findings of this report.

And secondly, express our concerns, our good wishes, maybe more diplomatic to ICANN to come to a right conclusion. Because there are possibilities to take a wrong decision which cannot be easily repaired afterwards.

CHAIR: Is that a response?

SHANE KERR: Actually, it's not totally unrelated, so I am ?? we have kind of the four different inputs into our root studies, right, we have got v6, signing the root, IDN and then turning root into calm. So, I think these have kind of, well they are all being thrown in because they are being considered at the same time. I think, quite obviously they have very different motivations. To be honest, I think we can discount v6. It's done. We expect more. But as someone said, we expect it to be limited. We have already got a plan in place for signing the root. We have a time line. I think that's going to move forward. So, I think we can't just ignore it, but I think as far as asking how and why we want to do this, I think it's kind of already been addressed. The other two issues are not really technical issues. They are political. Which is kind of related to what Rob was saying.

So, I guess since I am a technical guy, I like the first two and I am glad they are going forward and I hate the last two and I don't know why we are doing them. IDN I can understand because well, not everyone reads ASCII. So ?? in the Stockholm someone asked the question, did anyone in that room support expanding the root to two large numbers two new TLDs. And at that time not a single person was willing to stand up and say that they did, although I did meet people who said they liked the idea.

I want to ask a slightly different question: Does anybody here really like ?? I shouldn't ask it like that but I will ?? does anyone here really like the idea of a vastly expanded Root Zone with many entries?

CHAIR: Define "Like."

SHANE KERR: That's my question. You can tell me later.

CHAIR: We are almost 15 minutes into the coffee break. It's your coffee break anyway.

LARS LYMAN: Just to clarify the process. It is not, strictly speaking the case, that this report of the study team is going to go to the ICANN board for their review. There is a steering group, which is the body that was formed in response to ICANN's request that this study be undertaken, and that steering group is considering both this report and a number of other inputs and the actual recommendation and material that's going to be sent to the Board will come from the steering group, not directly from the study team report. Just a clarification.

CHAIR: Okay. Thank you.

By the way, can I have a show of hands of how many people have actually read the whole report? That's probably 25 to 30 or so, roughly, for the minutes. Thank you.

AUDIENCE SPEAKER: Jim Reid. My second concern is more of a layer 9 problem with the report in a sense. The report points out the need for having some kind of early warning system for alerting people to the fact that the existing processes are beginning to be under strain of breaking down under the load of all the changes that are going on. Now, what's been left unsaid in the report is who is going to have the ability to push that alarm button, obviously we need some kind of alarm button for it. Who is going to have the ability to push that alarm button and who is listening to it and what actions then can it take?

Let's take, for example, to give a little bit more of a hypothetical situation that could occur in real life. Let's say that under a TCP transaction start to cause problems D can the Root Zone operators say jointly and collectively stop because we are under a load here and if so, who is going to react to that and hopefully that going to make a difference?

DANIEL: To answer that one, at least from the status quo, some root name server operators have conclude exchangeses with ICANN about operational cooperation and certainly I can speak for Kay, saying that if we were under strain, we would let ICANN know in no uncertain terms because actually we committed to do that to them.

CHAIR: I am a bit in trouble. I am stealing your coffee break, if you have comfortable staying here. Just be silent. I'll just continue here and if I see the crowd disappear, we'll cut the quay which is now cut after Glen and then we have the wrap?up and and apologies to the online capture for the extra work.

AUDIENCE SPEAKER: The early warning system, that's one of the key findings of the report, and almost everyone seems to agree that this is an excellent idea that something we should really do etc., etc. I just want to make it clear that good as the early warning system or an early warning system would be, it is not the case that we have suddenly found the magic way of turning the bad, good question into a binary question. It's not the case that there is an early warning system that will suddenly go from green to red and we know we must do something. An early warning system is still something that's fuzzy. It's still something that go operate on different scales of well, scales of looking into the future and looking into how things change. So, even with an early warning system, we're still operating in a completely dynamic environment with fuzz in all directions.

AUDIENCE SPEAKER: Regarding that, a point of information, unfortunately at least for native speakers of English the term early warning is problematic. It sound like you are looking at the radar for the bombers to come over the hill in this case the early warning system doesn't just mean immediate problems, it means problems on the graph of discontinuity that Lyman shows whether 3, 18 and 36 months out showing problems. For ?? showing problems as we begin to realise they may occur. For an effective early warning system to be in existence, there is got to be a new order of coordination between IANA and their perception of the drivers of change in the Root Zone and the Root Zone operators and the people on the provisioning side and only with that kind of horizon can it make sense. Don't think of a day or a week or month it's too lates. Think of it in terms of months to years.

CHAIR: Thanks.

DANIEL: One more thing.

What I mentioned before, the agreement between ICANN, at least can say we keep other appraised an any developments that may affect the operational things and what Glen has just said is exactly that thing. We would expect that ICANN actually tells us, in no uncertain terms, what their long term plans are. And that is ?? and as we will tell them what our long term plans are and we both commit to it so I can commit to do that and it's all published. So as part of that, we would expect, and as I say this very clearly and very openly formally here, we would expect to be informed about the long term plan so that we can avoid discontinuity.

CHAIR: So I am not going to try to mistake wrapping up the whole discussion in main. I think we touched upon three very interesting topics here.

One is the relevant order and relative impact of the difference for suggested additional measures. Service levels and handling of TCP, measurement of TCP, operational queries on the Root Zone side and then this whole topic of early warning. How far in the future can we predict things to remain stable. This needs more discussion. I encourage everybody to read the report, if you have not yet done so. Provide feedback and continue that discussion on the mailing list, because I am sure that the team and also the steering group are still looking for feedback and will take that into consideration.

Now, that said thanks for the very engaged discussion, but you are not relieved. We are still having one remaining issue under any other business, and I'd like to draw your attention to this for another minute. If we get the video back.

JIM REID: Thank you very much. We were trying to get some kind of statement which we could send to ICANN to say thanks and endorse the general idea of getting the root signing. So while this session has been in progress, I have been circulating some text with a few folks to try and get something that I think we could all agree on and I think we have come up with a form of words here which everybody in the room could agree with. You might not be completely happy with one or two bits and pieces of wording, but I hope overall this is something that we are happy with generally. We agree that the idea of signing the Root Zone is a good plan. DNSSEC is ready for deployment and it's going to do good things by increasing the integrity of the global name space and we encourage progress words to signing the root. We want to make sure whenever the root is being signed as a wide circulation of this information, so that everybody knows that it makes sense to the general public and we have got confidence it's being done in a way that is not destabilise things and will also enhance the stability and the security of the Internet.

Now, if we can have broad consensus for that in the Working Group, I'd be happy to take that to the RIPE closing plenary tomorrow if Rob can find sometime for me there and have it endorsed as a statement of the RIPE community. First of all is this statement something this Working Group could accept and if so could I then try and pass it on to get a RIPE statement out of it? Yes?

CHAIR: We could actually call for a hum, which is a good idea. If you don't know this, I am asking for a hum in favour or opposed to this, if you are in favour, you can just raise your voice, depending on your comfort or discomfort, you increase or decrease the volume and then we'll take the opposite position. So hums in favour of this statement.

Opposed?

That is silence for the minutes.

JIM REID: Thank you very much, I am sorry for stealing the last few minutes of your coffee break.

CHAIR: Thank you very much, I'd like to thank RIPE NCC staff and audio staff for the report. Thanks for capturing the whole session and thanks for staying. See you in Prague.

(Applause)

(Coffee break)