These are unedited transcripts and may contain errors.
The EIX Working Group commenced on the 8th of October, 2009, at 4 p.m. as follows:
CHAIR: We are going to have a make a start on time, there is a lot to fit in so if we can get ready and start the meeting. Is the webcast sound all right over there? Okay. We are going to make a start on time, without further ado, three remaining updates from this morning's session. We are going to welcome Equinix J.P. NAT and Josh as well. While the slides are being set up, we are going to have the world premier of EURO?IX at this session and making your networks more a/PRABGtive for peering and learn more about Equinix carrier exchange project update on Celia:
Celia: I am working from Equinix. So Equinix is in the afternoon session because we are not a Europe end provider but global, so thanks for that, actually. I will start my presentation with what people know about Equinix. We are global data centre provider, we operate 45 facilities in 18 metros worldwide but we are global interconnection provider, we provide cross?connect services, also dedicated community of interest with financial exchange and so on. We operate 12?and?a?half Internet Exchange. I will explain later. But basically in every metro we have got data centre we can provide you with a connection to an Internet Exchange, whether it's an Equinix exchange or partner exchange.
This is sFlow graph of Internet Exchange traffic volume worldwide so you can see with peak traffic at 368 gigabits of traffic.
So what is Equinix exchange? It's a global platform. Basically, this means if you are a global network, you can get a service consistent across all the countries. But even if you are not a global network, you can benefit for all the global feature, 24/7 help desk, technical support by peers and metro performance information including up time statistics with latency information as well as multi?lateral peering exchange, support for /PRAOEUF vat VLANs and inknow /SRAEUtive functionality find a peer and invitations to the global peering forum.
So I am going to update on some partners exchange. LINX is available from Equinix data centre in London 4 and London 5. Lots of space there. Some people said there it's a bit far away but actually it's not that far. There is some new connectivity option to central London that has become available, if you have some question about them and interested please come and see us. Some /HRAT latency routes to central London as well as some Europe and US cities, avoiding the central London so some quite good to go to US and some from further.
Basically, to make the slide next generation ? of Europe.
DE?CIX available from Frankfurt. In Frankfurt we have got quite a lot, 39 carriers in Frankfurt 1 facility; 35,000 square metres in Frankfurt 2, that is quite impressive; Frankfurt 4 coming along with 9,000 square metres and 9 M V A of power. Not to forget to mention the ALP?IX available from Munich data centre.
Finally, AMS?IX, which is available from the Amsterdam 1 data centre, sorry, Amsterdam. Yes, we still have space available in Amsterdam. We have got Amsterdam 1 with 3,000 square metres and more to come. So don't be afraid to come to Amsterdam.
0 .5 partner exchange, why .5? It's because we come in age C IXP with CE R N, started in 2000, presented from Geneva 1 data centre and CE R N and Equinix and CE R N came to agreement in principle to extend to the Geneva 2 data centre which is coming early next year.
And update on Equinix exchange in Zurich. Some of you will know it as T IX located in Zurich 1 and 2 and expanding in Zurich 4, which is coming next year in Q 2. Synced with global standard. 62 connected members and peak at 7 gig of traffic.
Last but not least, paragraph /EUS Equinix exchange, one more. Available from Paris 2 and Telehouse 2. We use it first in 8 /600 ?? we achieve 3 gig of traffic peak. We have got 30 connected members.
Something I would like to the pricing, some people still don't know, fast gig port are free from Telehouse of Paris Equinix data centre, just install 300 it's 500 euro and 1010 gig port is free until the end of 2010, then you will be charged 1,500.
And that is it. Quite a lot.
CHAIR: Any questions for Celia? No. Okay then. We are ready for the JPNAP updates. Thank you.
KATSUYASU TOYAMA: Hi, this is, I am /KA*TS toy yam marks I will talk about JPNAP is Japan access point, commercial Internet Exchange in Japan. Our Internet layer base 2 X, we are not J.P. nick registry, we are not registry and Internet Exchange so please don't ask me for IPv4 address space because we have only /24.
And we are three area exchange in Japan: Tokyo 1 and 2 and Osaka. Tokyo 1 is biggest traffic, 180 gigs at peak. And also Osaka is 500 kilometres from Tokyo and these are independent network and means not interconnected.
And hot topic is Tokyo 1 expansion. Our new POP was built and Equinix T Y 2. Equinix is global all over the data centre company so if you are thinking about go into the Japan and especially in Tokyo and please think about Equinix and connect to the J.P. /TPHA*P.
J.P. /TPHA*P Tokyo 1 is in the centre of Tokyo and this green one, and T Y 2 is approximately 10 kill /OPL metres from the centre of Tokyo so expanded our network to this point. And using the dark fibres and we provided just the same as the service spec to the ISPs. Back end ports within optical switch and backbone servers are redundant and providing 10 gig and 1 gig E and T Y 2. Thank you very much and the traffic graph is added. Okay. Thank you very much.
(Applause)
CHAIR: Any questions? We are racing through the up dates. Is Josh in the room? If we run over time and miss the dinner it's going to be Josh's fault. Josh doesn't drink sew has got no incentive.
JOSH SNOWHORN: He means I didn't drink before this. So guys, I am going to present on our Istanbul Internet Exchange which is a different twist, so we are actually opening a new exchange. Just a little bit about /TER mark globally: Our major ex /KHAPBGS are in Miami and Brazil, most of the other data centres house other exchanges or hosting or co?low sites but if you have any questions about any one of these we will talk about that after. I want to talk about turkey right now. So there is no Internet Exchange there right now, in Istanbul at all and we are trying to create a regional hub we took space direct international access buts in Istanbul and all the big players there but the idea is actually to create an aggregation hub where we can pull in Jordan Syria Israel, Iraq and Iran. You have routers? We are going to have a force 10 S series switches two of them inside the facility, we shall going to start out with gig E because we don't see anybody jumping into peering too quickly. 10 gig is coming up and probably switches in 6 months. It is a decent co?low and well built site, we have been 1,600 square metres with another 400 we can build out to capacity. A quick shot through the pictures. /TER mark is is a flywheel based company so we don't usually use batteries. Security, camera system, FM 200 gas system because it is a small site we are able to do that. Quick summary and I am doing this quick because everybody seems to be giving me flag about going along here: SLAs creating a hub for the Middle East, if you are interested in bringing traffic it's going to be a great location. The ?? direct link to high /TPA* it's going to be a great place to do that. It's difficult in the Middle East to get and really turkey is fitting both ways, it's really European and middle eastern at the same time but best aggregation hub where all of these countries will feel safe going into it. Now I have a problem. What kind of name? We haven't figured out what to call. T IX, lots, it does sound like a blood sucking insect. B IX, a little too close. NA P, already have /TPHA*P of the Americas. It's kind of a sleepy name. Asia IX. There is a list. I have had T R ? IX. /T*URBG IX has been several people. Guys get to name an Internet Exchange. And I have got /EUS ticks and I IX which just doesn't work. Anyway names /T*URBG IX? I want you to look up /T*URBG IX.net, go look it up. Whoever can do it and somebody tell me what it says on there. Just having fun with these name thing.
CHAIR: I have looked it up and says we are going to miss dinner.
JOSH SNOWHORN:
AUDIENCE SPEAKER: Read out the entire web page "this is a test no flesh shall be spared one."
JOSH SNOWHORN: There is danger. Is it /T*URBG IX? Or IS T IX?
AUDIENCE SPEAKER: Another one that was PR IX.
JOSH SNOWHORN: We have Dominican Republic so we have the D IX.
AUDIENCE SPEAKER: T R IX.
JOSH SNOWHORN: I will give an announcement at the next EIX update and we will figure out what the name is and I will give you traffic updates it does go live next month, if you have any questions send it to me and I will help you out.
CHAIR: Wolfgang, who is going to give a talk on making yourself more attractive peering. Thank you very much for agreeing to the Jabber ?? you might have been forced to do those things but thank you anyway.
WOLFGANG TREMMEL: Hello everybody. I am Wolfgang Tremmel working for DE?CIX, in my former life I was a peering manager. This presentation was conceived as a part of peering to tore yell, we started this year for new customers. I adapted it slightly for this audience but basically it's just as ?? it's gone to several titles when I wrote it, like how to be a good peering manager, how to do peering procedures and we stuck then with this one, which is making yourself attractive for peering and the goal is of course to increase your peering success rate.
May I ask a quick show of hands here who actually represents an ISP, who does peer at one or other location?
(Show of hands)
All right. Thank you. Three basic rules for peering or being good: Be available, know your network and be professional. Be reachable and available. What does it mean? Your peers might want to contact you. They might want to contact you about the peering is full or you have packet loss or whatever and network attack, somebody might say hey, there is attack coming from your network, please stop it. So what we are seeing down there ?? out there is quite a lot of things that peering, peers, ISPs who do peering, the only thing they might have is a phone number for their customers or a phone number with robot in their local language. Does it sound familiar? A lot of peers, a lot of good ones out there they have a special web page for peering. Who has web page stating peering policy or of these ISPs? Four out of 50? Okay. And of course, that web page should be in English, as English is the language usually of course when it comes to international peering.
I have stated some examples here. Have a look, especially the first one, it's in German but they do have a English link hitting somewhere at the bottom. The peers should be ?? you should be reachable to your peers like your customers, meaning by e?mail, by phone, by whatever. Usually, the e?mail address which is used is peering at. By phone, well, as I just said, we have seen robots requiring a customer number. What customer number should the peer enter if he is telling you you have a problem in your own network and also if there is a person answering that phone, make sure she knows what peering is. And if you do IPv6 peering, be sure she knows also what IPv6 is. You can publish that information in various ways. You can put it into the RIPE record at the database. You have an entry for your AS anyway, why not use it? You can put it in peering DB. Who here has never heard of peering DB? That is good.
AUDIENCE SPEAKER: That is the opposite question.
WOLFGANG TREMMEL: Who has heard of peering DB? Check it out. It's free. I have an example here of entry in the RIPE database for AS 5669. The entry is fine, it tells you where to send peering requests and network problems, how to get copy information or network information. Actually I was putting that entry in there six years ago, I left that company five years ago, the entry is still there. It hasn't been touched for five years. So whatever you put into the ?? whatever you put there on information how to contact you, please, keep it up?to?date. It is worthless if it's five years old and I guess that all of these URLs here are e?mail addresses no longer work. Representing an Internet Exchange, of course the Internet Exchanges have to know this as well. They might be contacted by other customers who are not able to get in touch with you so you can help them but if they are not up?to?date we can't contact you either. We do publish a list of operational contacts for all of our customers, but if you don't update us when something changes, it doesn't make it on the list. Know your network. Who does know his own network really good? Usually, it should be the peering manager, he should at least know where his peering points are and where he peers. The NOC manager should know where all the routers are. And both should know what they are announcing. You peer with these guys, I don't want to do blame and shame here so I err raised the first digit but it looks like that somebody is just dumping all his internal routes into the internal BGP table so when I was peering manager I had a pretty open peering policy but one of the reasons I denied peering if someone was just doing deaggregation on a massive scale he didn't get any peering. Make sure your BGP announcements are really clean. Aggregate. It doesn't have to have 100%, also somebody might disagree with me here so there might be reasons that you announce one or two more specifics but do not announce all more specifics and especially do not try to announce anything smaller than /24 if you talk about IPv4. Also, most of you might have special customers with special needs like Internet cafes or gambling sites or whatever which attract more traffic than they should do sometimes, know who they are and how to handle them if an attack is coming your way. Be professional. What does it mean, being a professional peering manager and handling peering in a professional way. How many of you peers here do have an open peering policy? Okay. How many of you have a restricted or policy based peering policy? So please keep your hands now up. Who of you have published this peering policy? All right. That is food. Better than I expected. About five. Yes. If peer talks to you, and says there is an attack, help me /WHARBGS do you do? Do you do the correct choice? Talk to your customers, shut down your customer if he attacks someone or block the bad packets or do you just say "go away, I don't care" I have seen the first one cried often during my time. Document your peerings. If you have a look at the AS objects in the RIPE database, we have seen this morning, I was saying in another presentation where somebody compared the peerings he saw on his exchange to the peerings documented in the RIPE database. This information is usually hopelessly outdated. Who here really keeps up his AS object with the peers he really has? Four, five, six, seven, eight, out of 50 people peering here. What is up with you? This is not professional. Also, if you announce may not tenses on public mailing lists is it on an exchange or if you have your own mailing list, do it in English, please. Don't send word attachments in your local language, never, ever. We see this every other week on the DE?CIX ?? not on the DE?CIX mailing list because we block attachments, but on our internal mailing lists. People send something us a word attachment with the subject "please open." Another thing: We finally educated our customers not doing that any more, the last point do, not announce every new prefix you are announcing, either publically or privately to your peers. We know because there is a sing like the RIPE database. This is not for the guys with the restricted peering policy. Like I said, please, respond to every peering request, even if it's a guy you Noelle never peer with, just be polite. It's nice to be polite. Also, if he does not fulfil your requirements, give him a nice "no, we don't peer with you" don't say go away and buy transit. I have heard that quite a lot of time and I would never buy transit from somebody who is not nice to me. And if you denied and tell them why you decide them, so they can improve, they can grow and even, next year, if they fulfil your cry tier /TKWRA, you might peer with them. (Criteria).
A word about peering contracts: The usual disclaimer here: I am not giving any legal advice. I have experienced the topic of peering contracts quite often. At the beginning when I took over the peering manager role I forwarded each and every peering contract I received to our legal department. It took them three weeks usually and they always had changes. I have no idea why they never can accept any contract coming from outside without changes but that is the way it is. I am sure, I see people smiling here, I am sure everybody has experienced that. So to make your leave easier, there are two things you can see: One is on the slide and the other is not; the one that is on the slide is ask the other party to sign your peering contract because then their legal department has all the work of changing the small little words and making the life of the other peering manager horrible. It sometimes works, it sometimes doesn't. In case it doesn't work, why don't you tell the other party, oh, yes, it's nice but your peering contract but our legal department is so overworked. Why don't we just set up the peering. We will take care that the peering contract is signed by we can start peering right away. And in 50% of the cases, nobody ever again asks for us to sign this peering contract. So, it gives ?? peering contract in itself, they are not a bad thing, they are not a bad thing, I am not saying that they are really bad but they don't help you to get the depeered or not to get depeered, I have never heard of a court case based on a peering contract that is not with small provide /ERBS or medium?sized providers but I think they are quite useless. If you want to stop peering with someone, just do it. It's always a bail out clause in the contract so you can always get out, but they just keep the legal people busy and don't help you in your day?to?day life. Do you have any questions?
AUDIENCE SPEAKER: /A*RPB Hughes. Thanks very much for putting this together. It's nice to have a reminder and love to see it in NANOG as well. I wanted to emphasise a point that half a slide in there that said use peering DB is quite important. Please, please spend sometime there, it is the only neutral open place to update information about your locations and IP addresses, but we are /K?RPBly working on quite a few tool suites we will release to the public, there are a few we are working ton automate IRR updates as well as ?? as well as selecting contact information to e?mail your peers rather than e?mailing the entire peer list. Some of the reasons people don't do are because it's not easy and we are going to try and make it easy but the data has to be accurate in peering DB for this to work. Please take the time to update the records. One notable point on v6 every network I have seen writes their v6 address in D /TPWH?FPLT a different way and it would be nice if everybody would please take the IP address that you get and excess it and put it in correctly instead of random numbers of 0s and colons.
WOLFGANG TREMMEL: To answer that, there is a second part of this presentation, give this presentation internally which is called peering tools and it focuses very much /O on peering DB.
AUDIENCE SPEAKER: Do you publish tools publically?
WOLFGANG TREMMEL: I can send you the presentation the next time I give it, because I always update it every other month.
AUDIENCE SPEAKER: Fantastic thank you.
GERT DORING: Be professional about things. Don't do your public peering negotiations on public mailing list like the exchange point lists.
WOLFGANG TREMMEL: Good point.
CHAIR: Thank you very much indeed. So in summary be nice and use peering DB.
We are going to have the world premier now, we have a film, so popcorn at the back of the room. Do you want top introduce the film.
Cara: A little bit of background to this. About two years ago, actually, we had a EURO?IX forum, it was in Amsterdam, and a couple of people were discussing that there were, they had been watching a movie called wariers of the net and we were saying to each other hey there is actually not something on?line, a movie, that explain what an Internet Exchange actually is. And let's see if we can set that right. So we contacted these people of /WA*UR wariers of the net and they gave us well huge figures of how to change that movie into something that would explain an Internet Exchange, so we thought of something else, and wrote out at competition for professionals and amateurs in film making to develop this movie for us and we started this out about a year ago, I guess, maybe a little bit longer, road out the competition and we got about 20 responses that were interested in developing it, then from those we chose a subset and we, after the first ?? after the first phase, we clearly had a favourite already and in the end this favourite won and this is the end result. And it's actually I am very happy with it.
Video)
Cara. So. This was a EURO?IX effort and it's been supported bay number of the EURO?IX members that did this as a separate project and it's now finalised but everybody within this community can use it as long as it's not for commercial purposes. There will be subtitles to it on the YouTube version. Actually it's already uploaded currently, so if you Google or YouTube it, I guess, it's called "the Internet revealed" and if anyone is interested in translating a version, let me know.
SPEAKER: I wanted to add a bit more that Cara didn't want to talk about. At the end you see all those flashing websites and I don't know if anyone seen fight club before and knows about that guy that used to put these images in, these guys had a go with that so we had to ask them to take some of the pornography out of that so it was a bit checky of them. On the point of translating, I did upload subtitles even if you do YouTube in the Internet revealed and you click, if anyone knows thousand do the /SAOUB site he will thing. Now it's in English. If anyone is keen to help us out with sub titling in every language, please let me know, I will send you the text and you send it back to me.
Bill: It would have been really good if you guys had run this past the regulatory affairs officers because there is a couple of things will make real problems if they government governmental audiences, the idea that peering is shoe sharing of workload and the idea that peering involves anyone giving anything for free to anyone else, both of those are pretty substantial mischaracterisations of the peering transaction and both of them are asking for leg tree intervention where none is needed.
SPEAKER: Okay. Any other questions? Okay. So, I don't know if I get any help here in getting my presentation up. I will introduce myself first. I am Serge rad Vic of the European Internet Exchange Association. So anybody EURO?IX is the social of Internet Exchange points originally we only had European members but since 2005 we expanded to invite IXPs from every corner of the world and that is pretty much happened. Today we have got 48 affiliated IXPs, 39 of those are based in Europe at about 25 different countries but we also have a member IXP in Africa, we have got about five in Asia and three in the States, so it's really becoming quite global. We also have some /PAEUT reason tos that not only financially support EURO?IX but actually get involved, they come to our meetings and get on the mailing lists and they really part of our community as well.
Now since the last meeting, in Amsterdam in May, we have had four new affiliates join, two of which came out of Italy, F VG IX and also the second Luxembourg Internet Exchange has decide today join EURO?IX and the latest addition was NL ? IX of the Netherlands which is well?established IX that has about ?? between 20 and 30 locations in the Netherlands. So with the two additions from Italy we have got six member IXPs that are based in Italy and there is a seventh one who is coming to our next meeting who might become a member.
One thing I try and get the ?? our members involved in is an automated effort to grab the statistics. What you see here is from 47 IXPs. Now, I said there was 39 in Europe, there was about 32 involved but we have got IXPs like NetNod who have five separate locations, NIX Norway has a few and so on so it represents 47 separate IXPs in Europe and this 2.9 terabit traffic makes up about 90 percent of the 121 IXPs in Europe, so our members pretty much make up the bulk of the European IXP traffic. Looking at Europe as a whole, this is ?? I am in contact with probably in, good contact with around 80 to 85 IXPs in Europe and I am able to scape a few others so I think there is estimates on around 10 to 12 of the 121 IXPs so I think these figures are pretty accurate. So my estimate is 3.15 terabits per second is what is going peak across Europe at any point in time ? well, at the peak point in time. And that is about 60 percent up on 12 months ago or over a terabit so it's still moving. I am going to go into the traffic a little bit deeper in a minute. But what I wanted to do is also compare the European statistics to some of the other regions. Now admittedly, I am in very good contact with probably more than 90 percent of the European IXPs, but I am either not or I don't have the ability or time or resources to be in contact with the rest of the world so the other regions that you see; USA for example I think I look around seven or eight IXPs around the US so while I feel it's quite representative of the trend there, it's not as accurate as Europe and the same with the other countries and regions. So anyway, Europe and the US still seem to be head to head over the last 12 months, August to August I am talking about here, around 50 to 55 percent. Asia always seems to lag quite a way behind and they have done that again this year. And Brazil are over 100%. The P TT guys there have got I think nine separate IXPs, unconnected IXPs in Brazil and they have got some good traffic stats so taking a global average, it's around 50%. So what about comparing 2009 to 2008. I looked at the end of January in 2008 to end of September and now the same period in 2009 and as you can see overall, over that nine month or so period where still looking at around 20 percent, and the graphs looked something similar, the big difference is the end of summer increase in traffic, the take up. Usually, this occurs around the end of July or at latest end of August but this seemed to happen later this year. You see that in 2008, you see there is around 15 percent increase at the end of July in 2008, sorry end of August in 2008 which is compared 23.85 this year so it was a really late take up /TPWHU jump here that you see in 2009 is about 16?and?a?half percent so it's really caught up. I am still predicting around 50 to 55 percent growth in 2009, so as you can see, in the next three months, it's going to be quite a bit of growth in the IXPs in Europe.
For the first time I had a bit of look at making some projections and predictions on a longer term. I had been taking pretty accurate statistics. These are based on 36 IXPs that I have been looking at since around 2003, it's not the whole of Europe but again I think it represents the IXP scene in Europe quite well. So, putting annex /POEPBGS trend line against this and going lieu to 2014 I am going to put a few euros on the fact we are looking at aggregate peak traffic at around 40 terabits in Europe at September 2014. So I will update you next year and tell you if I still believe that is the case. I had a bit of a chat with a couple of the bigger ISPs, AMS?IX for one, just comparing the figures I had and their figures. They were a little bit higher than this so I think the 40 terabits isn't ?? will probably be easily achieved. We will see.
So who is causing all this traffic? I now calculate there to be around 5200 participants at the 121 IXPs in Europe. Every year I get in better and better contact with the IXPs, they also produce better information on their websites, so while I say there is a 15 percent increase over the last 12 months, you could probably take 2 to 3 percent error factor off that, because now I have got a little bit information and more in my database so it's probably about a 12, 13 percent increase, and this is pretty much been the case for the last four or five years, so the participants are still turning up new participants are turning up, so around 3,000 or so of those are unique ASNs, and there is at least 780 ASNs I know that appear at more than one IXP in Europe and this is probably around 15 percent increase over the last 12 months and from what I can see, there is at least 29 ASNs that are peering at 10 or more IXPs in Europe and this number is growing. Last year it was around 20 and the year before a little bit less than that, so, yes, those numbers are also increasing. So these are just the European figures by the way. So these 5200 participants are accessible through the EURO?IX ASN database. I guess it's ? dish shouldn't say it's an alternative to peering DB. It's a little bit different. The essential difference here is the ASNs are collected either by myself /A*EU go through a lot of the websites, or our members, our 48 members update this database themselves, so for what those guys update the information is pretty accurate. For the rest, it's as accurate as the information that is made public by the rest of the IXPs, so if you go to our database, either EURO?IX /resources or you have a look on our home page, you will see the ASN database. If you are looking to see where a particular ASN turns up you can plug it into the search query. You can do the same for a name, what different AS inspects that is going under and list ISP and see what /THAEUPLS are there. It's pretty complete. Another little feature I added, if you do click on the ASN, then you will go lieu to peering DB. So that is how we have hooked up with each other.
(Applause)
And they pay me a euro for every time it goes through. So a little bit of news. We hold two meetings a year that we call the EURO?IX forums. They are invitation only meetings. They are there for our members and patrons but we always invite prospective member IXPs to join us as well. We usually have 100 or so representatives, I am expecting this time around 40 different IXPs to be at our meeting, we have already got 37 registered so I think we will get our record of 40 and we talk about a range of issues from commercial, technical, the route server issues are going to be a bit of a focus this time around, also regulatory. Anything that IXP specific and focused we will talk about. We invite a couple of ISPs along to give their views of what they think of IXPs but it's pretty much IXP, /SREPBTDers co?los only. If you are interested in knowing more about it or feel you should be invited come and approach me and I will let you know if we can accommodate you. Our resource section, that is EURO?IX /resources, you will find a few things there. I have just published, well, at least I have published it to our members, the EURO?IX annual report on European IXPs, that is just a whole about 50 pages of the IXPs in Europe, traffic trends, customers, switches and all that sort of stuff. They are just reviewing it at the moment and telling me if there is any mistakes. I think we have got another two?week review period and they will send it back to me and I will make it public. I have got a few people on mailing list, if you are interested contact me and go to EURO?IX /resources and you will find it there in a couple of weeks. We had a crack at IXP BCP. Working on for a couple of years. This has been made public, if you go to resources you will find the link to this. And I have also, through a push from the couple of our /AEUGS member IXPs, tried to make a list of extensive Asia Pacific IXPs, I have probably got most comprehensive and up?to?date where you will find 121 different IXPs listed and links through to the traffic data. I have now also a/TEFRPBed to do that for Asia Pacific region, obviously I am in less contact with those IXPs but trying to improve that so have is a look at that list. If you know some additions or IXPs that don't exist any more let me know. A little bit more news: I won't say how much about it but we are planning to go even more global than we are now. As I said, since 2005, we have already had members from different continents. There still needs to be discussed and decided by our membership at the next general meeting so I am not going into too much de/TAEUFPLT I will probably give you a better update at the next RIPE meeting in Prague. The EURO?IX route server initiative, you have heard that mentioned from a few of the IXPs. This is something that seems to be getting quite a bit of buzz at our community and a lot of IXPs involved and I believe that Nick is going to give an update on that after this so I am not going to go into it and the film competition and we have a winner which you just saw so obviously no more information there.
I think that is about all I had to update you guys with, so I don't know if there is any questions?
CHAIR: Any questions for Serge? No. I think you have been let off the hook.
(Applause)
Okay. Now, Nick is going to update us on the status of the route server refreshment project or improvement project. Standing in for John
NICK HILLIARD: Thank you very much Andy. John was originally planning to give this talk but unfortunately fortunately wasn't able to come to RIPE 59 so I have agree to give this talk instead of him.
So, just brief background. Yes, route servers have become enormously more popular over the past couple of years in the IXP community. As the number of IXP participants has grown very substantially. And pretty much until a couple of months ago, the only realistic BGP deem encould thank could actually provide fully functional route server functionality was QUAGGA. I gave a presentation at the last RIPE on why exactly this was so. It's got to do with multiple ribs and views and all that sort of stuff, so the problem is that QUAGGA was not really designed with this sort of thing in mind and it has very substantial scaling problems which some of the larger Internet Exchanges have found out to their detriment. The problems that we have been seeing have been deem ens crashing, session being re?set, rooting flaps, all of that sort of stuff and it's not really a good thing to present a service like that to your customers. So, a bunch of us who are all members of EURO?IX just decided to get together and to try and sort out this problem. We had a bunch of options: There is an all of lot of pent up angst and verging on hatred of QUAGGA in the IXP community, a lot of it quite well justified. We had a couple of options in /TERPGS of alternatives. The first is (terms) BIRD, which is a development by NIX?C Z. Andrei is in the room somewhere or other, Andrei does cool stuff with BIRD, really lovely configuration system. The project had been sort of very much frozen over the past couple of years but the recent activity in terms of route servers you know sort of kicked the guys in the Charles University into action and it is now in operation as route server in one or two exchanges around the place. Open BGPD was one of the other options. It's been on the block for a bit longer. Everybody knows it, it's part of the open B S D project. It didn't have multiple ribs and AMS?IX have funded development of a multiple rib view system for open BGPD and I believe that they are going to be implementing a live open BGPD based route server system fairly shortly.
AUDIENCE SPEAKER: Next Thursday.
NICK HILLIARD: It didn't really seem to be a whole lot of other options a couple of months ago when we started looking at this. We had the option also of sort of just trying to dissect QUAGGA, seeing what the story was, you know do you do a general thing and try and get the source fed back into the QUAGGA /TKPW*EUT repository or fork off a branch for the ISPs? It wasn't clear what the best thing that could be done. One of the things which was causing really serious problems was for QUAGGA, based ISPs was every so often fat fingers would cause extra prefixes to be injected into the route servers and, you know, everybody does it, the large carriers as well as the small, the largest carriers as well as the smallest, and everybody else in between. And it caused QUAGGA an all of lot of trouble because sessions would get re?set if you had max prefixes and if you didn't have been max prefixes set, well, then, you know you'd send the CPU spinning because you would have all sorts of problems and delays and stuff like that. Anyway, there was all sorts of problems there. At INEX we took a decision on day one to implement inbound route filtering use and that has been a major win in this regard. We can do that quite easily because we are very small; larger IXPs cannot do that easily because they are very large and when they try do that sort of thing they run into all sorts of scaling problems that we simply don't have. So this was a bit of a problem for them. The other option of course which was being talked about was well, look, hang the whole lot of them and write our own deem enfrom scratch. Really tempting if you have been working with QUAGGA long enough.
So, yes, EURO?IX seemed to be the best sort of forum for doing this. It was a good way of just getting all of the interested IXPs together and we figured, we made the astounding realisation that unless somebody was actually from the IXP community was actually going to go out and fix these problems they weren't actually going to be fixed. It seems quite remarkably dumb in retrospect, but hey. So what have we done? We have established route server database which I will show you in a few minutes, a useful thing. We have virtual Working Group of remote participants. We have gone around to all of the EURO?IX members and beaten on the door and demanded money, preferably upfront, use fivers. We published a request for proposals for fixing some of the more glaring problems with QUAGGA and we have evaluate add bunch of responses to the RFP and we have accepted a particular QUAGGA improvement project and of course, the fission thing now that we have realised is having done this, that testing is the major problem, if you run QUAGGA on its own it seems to be okay, it's only when you start sort of connecting it into, I don't know, 100 or 150 route server clients that it really starts burping in a rather unseem Lymaner and we have also considered funding other non?QUAGGA projects outside ?? this is in addition to the funding that AMS?IX has made available to the open BGPD project. This is the route server database. It's very quick and easy really but it's remarkably useful. It's hosted on the EURO?IX website. It's not actually available to the public but, all of the EURO?IX members can see what what earn else is running so here are a few of them. You can see that people are using CVS investigations and head investigations, for open BGP that is quite useful. The project to fix QUAGGA, the most important aspect of this we figured was, well, you know, if we fork it off, it's just going to suffer bit rot and then it will die so it's very important strictly for us to get this (strategicly) into the mainstream /TKPW*EUT repository. We are currently working with a developer and doing an all of lot of pro filing. Just by way of figures and all that sort of stuff, in the first week he took the startup time for reading and procession one of the IXP configurations from 15 minutes down to 52 seconds, which was quite good. It's got to do with data structures. If you use the wrong ones in your code, the performance will be complete trash. And we have identified a lot of areas in the code where this was the case. So, hopefully, once we have got the testing rig up and running, we can then start doing some really serious performance improvement.
We also received a rather interesting proposal for a completely new route server from a ?? yes it's from an existing code base, it's not a full BGP deem enor anything like that or fully featured for what it does but it claims to be very, very fast inseed. And EURO?IX has agreed in principle to fund a proof of concept which is to say that it should be able to read and process a configuration file very fast indeed and should be able to handle a specified number of route server clients and do all of that stuff really fast. The proof of concept is IPv4 only, 16?bit ASN only. One the proof of concept has been agreed and we are happy with it, the things like IPv6 and ASN 32s and all that sort of stuff will be implemented. They are relatively easy, it's not a maker problem. The working title of the project is bow depart and I have spent about 10 or 15 minutes looking through my e?mail last night to remember what this stood for but unfortunately I can't so (B O G A R T) maybe at the next RIPE meeting I will manage to find it. So this is where we hope to be relatively shortly. We should have a much improved QUAGGA. It won't be perfect because QUAGGA has some design issues which will require an all of lot of fairly deep surgery to fix. We have BIRD, which is working and I believe working very nicely. We have open BGPD which is showing an all of lot of promise and B O G A R T and they are all fit for multi?lateral peering deployment or at least should be relatively shortly. There you go. That is what we have been doing. Any questions?
CHAIR: Again no questions. Okay.
NICK HILLIARD: Thank you very much.
(Applause)
CHAIR: We have got a couple more talks now, we have got Maarten from the AMS?IX who is going to describe how to rebuild a plane in mid?air, presumably with an Internet Exchange in the middle of it afterwards.
MARTIN PELS: It's actually not my title, I think mike came up with it first, but I am not really sure. Anyway, what I would like to tell you is basically how we went from this to this without this. So we started off with some preparation about a year ago this whole thing started and we started to look into VPLS so the first thing we did was debug pro/KAEUDZ and VPLS code. It was horrible at that time, I think we found about 30 bugs, small to serious. And we test this had extensively, we went to brocade a couple of times in US to test in their lab and they were very helpful and actually resolved everything and added a couple of features that we needed. So then we built some new software on the old platform we had some software that reacted on SNMP traps generated ?? because we are going to disable VSRP we needed new software so we did that. We developed configuration tools, new platform has a couple of 100 LSPs, a couple of 100 and probably run into the thousands at some point so we described the whole network in XML and we generate our con filling rations from that. Then we moved our access switches for 1 gig E customers behind our access switches for 10 gig. Customers. We replaced some hardware because we were running with some brocade switches which do not support MPLS and /TPAOEUPLly we defined migration scenario to do the actual migration and merge the platform without customer impact.
So, first thing we did was /KEBG the access switches for 1 gig E customers behind the photonic switches we have, which are electronic batch panels and the problem we had is that we wanted to migrate one half of the network to MPLS first, and the other half later. So we we couldn't use VSRP any more. When we moved these switches behind the photonic cross?connect there would be no more physical connection between the two core switches so there would be no way that these switches could talk VSRP to each other. Then we de/AEUBLD VSRP, we had to install a different kind of software, which basically reacted on link flaps, when a link flapped in active network, SMP trap was received by the software and failover topology was initial /TKWRAEUTed and switches swap their connections.
So yes, we did that, we moved access switches behind photonic cross connection so act adds normal customer. We migrated one half of the platform to MPLS. We started with load balancing over two core switches, just to test if the whole load balancing thing would work all right. And we did that for about a month, I think it was five weeks in total. And then we did the second half, so then we ended up with two separate MPLS platforms. Actually, so at this stage, there was in August I believe we had two separate MPLS platforms, both with two core switches so that already solved our scaling issue because we could scale to all the ports on both core switches, but we also wanted to fix the problem of doing complete platform failovers because that introduced too much problems. So, what we want today do is merge both platforms into a full platform in one single active platform. Of course, we had to do this without any customer impact. We needed to do all sides together and we needed to keep the time between having a completely working platform and a completely working backup platform as short as possible. And so yes, we had to come up with something to do that, all in one maintenance window or at least something close to that. So this was the initial situation. This is a close?up of our core infrastructure. We connected all the access switches to a large photonic cross?connect at the core site and from there all the links go to the different core switches. We have a red and blue network and we start with the red network active. What we did then is reduce the capacity on the blue network to 50% of the load so that we could take out half the links. We then have to wait until about 1:00 in the morning until the traffic levels over platform went down enough so that we could actually run on this reduced blue network. So yes, we really had to sit and wait until that finally was the case. So that was 1:00 in the morning and then we swapped over to the blue network. From then on, we had about 6 hours to bring the blue network back to full capacity because at 7:00 in the morning traffic starts to pick up again, people are waking up so you really don't want to do any meant /TPHEPBS any more then. So what we did was we took (maintenance) half of the links from the red network and swapped them around on the access side. On the core photonic cross?connect we did some reshuffling web interface, this goes there and that goes there, no plugging cables on site. So we added all of the ?? half of the links from the red network to the blue network, added all the other speeds and MPLS paths and finally added the LSP to the VPLS instances on the blue network and we can all do this trance /HRARPBTly without any customer noticing. So at 7:00 in the morning we were pretty much done with this and we had a blue network at full capacity again and, at the same time, we did same on the red network, we took half of the links of the blue network that we took out earlier and added them to the red network, put on MPLS path, LSPs, etc., etc. And what you see at the bottom is that from each photonic switch there goes a link from the blue network to each other switch and from the red network so we basically split the original aggregated link in half. So at this point, we had two completely redundant, well we had still actually two networks, but now they were all using all four core devices, and we didn't have automatic failovers yet. We could do manual failovers at that point, so if there would have been a problem, then we could have fix that had just a little longer than normally, but at this point it was 7 a.m. already so we really couldn't do any more intrusive, potentially intrusive maintenance, so we went asleep. And then the next day, we started adding LSPs from the red network to the blue network, well previously red and previously blue because we are now building one entire network so if you have the two switches at the top, we add LSPs and MPLS paths between those switches so that all switches ?? all access switches were active at the same time. And then we enabled a tool called P X CD which works the same as the previous tools reused, it reacts on SNMP traps but this on failures instead of link failures and finally we distributed all 10 GE customers over the different access switches so all were active at the same time, and that is where we are now. This took about, this took two more nights of playing but nobody noticed. So was basically it. We did a successful migration of the platform, the only thing customers saw was one or two link flaps and most people, for most routers that means no flapping BGP secretaries, for some it does but also had that with the old platform with. The total time for this project was less than 11 months, we started end of October last year with our first conversations with brocade and early September this year we did the actual migration or we finished it. The whole merge of the platform was done in three consecutive maintenance windows and we had about 12 hours where we did not have a completely full network at full capacity but it was always had enough capacity to handle the load at that point. And of course, we had no customer impact. And well, the main thing here is that we could really not have done this without our /TKPWHREUPLer gloss photonic switches because it would simply have been A, too much work to physically replug everything, we would have needed a lot more people, and B, things like a failover you can do that a lot faster by just instructing everything at once to repatch than by doing it by hand. So that is it about the migration and there was a a talk did I a couple of days ago about operational problems we saw on the platform. It was one operational problem that I didn't mention yet and that is what we call our boxes now. We used to call them switches but, well now, they talk MPLS and do routing and do we call them routers now? I don't know.
AUDIENCE SPEAKER: Routeers.
SPEAKER: Oh shut up you stupid /PWREUT. They do something with labels, so I thought more like something like a label writer or ?? you get a network like this. Anyway, questions?
CHAIR: Thanks very much Maarten. No questions at all? Everyone must be super hungry already. Okay. Thank you again.
(Applause)
Last scheduled set of ?? from Remco who is going to be talking about Equinix new carrier exchange product. Thank you.
Remco: I am being forced to stand here by powerful people in smokey back rooms. I didn't plan on doing this presentation and like you, I'd rather be somewhere else right now. But this is not the secret Working Group, thank God they only show it once each meeting. So instead I am going to talk to you about our new ethernet exchange platform. What does it do? And it call came into play because of this press release, as you can see that was Tuesday, in the parallel but not entirely disconnected world called ITU we made an announcement in Geneva, that we are going to develop our first carrier neutral ethernet exchange and of course, that raised a few eyebrows and a few questions in this audience, what the hell that was, so I will try to help you out there.
First, a brief history of carriers ethernet. Warning here, there is some artistic licence, I am /SPHAEUBGing /OPL very all of shortcuts, don't come running to microphones if you feel I am not representing history entirely 100% correct but here is an attempt. So back in the 1990s, carrier and ethernet fit together like honest and politician. Carriers wanting to have nothing to do with ethernet, kind of immediate observinger technology, there is a quote I remember from those days "you can't even route or assign addresses," interesting. Why that was important I still don't know. Which at the turn of the century raised the question why are our customers asking for this stuff these days. I have no idea. It's a poor technology why would they. Out of money, the interesting question why do these people pay ten times less for their equipment? So after 2001 the metro ethernet forum came into life which was "let's redo all the carrier things using ethernet as a protocol" instead of what you are having right now. How did that develop? First thing they set up a new standards body as they do. Second thing they did was deal with the quality aspects, so they introduced the world to fantastic things like two bit ethernet, when they had that sorted out, they dealt with the UN I or as we call them, CPEs, which is how do we interface with the customer, what do we want and what do they want. That has been pretty much sorted out in fantastic protocols called die and gasp. It's interesting stuff. The next step ethernet forum is at right now is dealing with N N Is or /TKHOU carriers talk to each other using ethernet or delivering ethernet. And so what is this N N I anyway? I took this off Wikipedia /ARBGS network to network interface ?? read the web page. It's a lot of complicated language for, big surprise, peering, not necessarily settlement free but it is peering, nonetheless. So what is an ethernet exchange, it's not an Internet Exchange using eater net. Ethernet is relatively easy, carrier Heather net is harder by anyway /TAOUFRPLT inter?domain is a challenge and multiple inter?domain carrier ethernet is complicated.
So the ethernet exchange is an effort to scale N N Is so links between networks from a one?to?one solution you really must envision that carrier A has a /TPWHOBGS a rack and there is a cable going to another box in another rack from another carrier and those two boxes are used for connecting those two networks and not used for anything else. The ethernet exchange is an effort to go from that model to private interconnect to share a platform to ones we have been used for a decade and a half. It's not just forwarding packets, that would be too easy; there is a lot of stuff in carrier ethernet that you need to look at, you need to translate, identifiers like Q and Q, you have quality definitions, class of service bits, DSCPs, which not all carriers might treat identically, you have got SLAs to take into account, what service am I provide to go what customer and what guarantees am I giving and what is the other carrier doing etc.. so an absolute metric boat load of standards comes into play, DSCP mappings, numbers T E VPLS V L L. And this is hard stuff. And even the carriers don't know, really, yet, what they want from a service like this, so we are doing a pilot right now together withical /KA tell, they have a very complicated box that can do super complicated things with ethernet packets. And we are giving it a try. We are trying to establish what the community wants out of a service like that right now. One of the things I can say to you, and this will probably surprise you, is that the traffic that goes on between carriers on the ethernet is a lot less than would you think, so in ?? even though all Maarten just had a presentation about being able to ? 12810 gig ports altogether this is the world where 1 gig interface still counts as a big interface and with that, that is really it. There is nothing really more to say seeing as it all still work in progress. If you have any questions, feel free to ask?
CHAIR: Thank you. The mike is looking a bit empty. I have got one. So what do you think the feature, do you think there is a significant feature in this particular model for Internet work, interconnectivity from today or do you think it will still happen across hard ? /PWOUPB trees as we normally do it on Internet Exchange?
SPEAKER: Well, I think we will see a couple of separated worlds for a while. You could perfectly run an Internet Exchange on top of an ethernet exchange, as in the same way could you deliver milk in a Ferrari, so you could do it but it makes little sense to do so, if you look at port density per port of the switching platform required to do a carrier ethernet exchange that has very significantly different price point, complexities, responsibilities for the operator of such an exchange and also quite different so I think well, the whole concept as it is is as yet to improve, and the /*ETer net forum has published that. They probably would like some neutral third parties to sit in the middle between carriers and facilitate these exchange N N Is but what that /K?BGTly will amount to is as yet unsure.
CHAIR: Okay. Thank you. Right. We are currently 30 minutes ahead of time aren't we. We are bang on time. We have one minute ahead of time. Thank you very much for attending EIX, I hope that was useful and we will see you at the RIPE dinner later hopefully. Thank you.
(Applause)