1997.04.10 REPORT OF APAN TRIP TO NORTH AMERICA ------------ AIMS OF TRIP ------------ The trip was proposed by Prof. Kilnam Chon to get/exchange information on design, operation and management of high performance networks. The goals of this trip includes: - to obtain management perspectives, - to obtain first hand knowledge on large ATM network operation, - to see cooperation between ATM experimental network and production Internet for research and education communities, - to hear the experience of XP management/operation, - to improve "APAN Network Design Memo" with the comments of experienced engineers in North America. ------------ PARTICIPANTS ------------ See Appendix 1. -------- SCHEDULE -------- 03/14 vBNS operation at MCI (Reston, VA, USA) 03/17-18 Chicago TAP (Argonne, IL, USA) 03/19 CANARIE (Ottawa, Ont, Canada) 03/20 Teleglobe (Montreal, Que, Canada) ---------------------- PRESENTATION FROM APAN ---------------------- Overview: See http://apan.net/documents/apan/index.htm Network Design Memo: See http://inoc.imnet.ad.jp/APAN/APAN-Design/ -------------- RESTON MEETING -------------- 1. vBNS operations - MCI operates vBNS for 3 years. - vBNS PVPs are over MCI's commercial ATM network. - vBNS ATM switches have full mesh PVPs. - Testnet was upgraded to OC-12 at first. It took only 3 weeks to begin the upgrade of the production network, vBNS. - Feb., 1997, backbone will be OC-12 with "Vector" ATM switches (Nortel), leaf networks use OC-3 with FORE ASX-1000. - Netstar Giga Router for IP. Currently Cisco cannot handle OC-12. - 7 ways to connect to vBNS (1/2) OC-3/DS-3 to vBNS Router (3/4) OC-3/DS-3 to vBNS ATM (5/6) OC-3/DS-3 to MCI's HyperStream ATM switch with a PVP/PVC through to the nearest vBNS Router (7) NAP (same ways will do for GigaPops) - NSF' AUP has been relaxed, and 100 universities will be connected; vBNS has little traffic and doesn't suffer from packet loss, but it is difficult to estimate the packet loss in the future configuration. - Use of BGP Communities to implement NSF's AUP * Primary Peers (SCCs and New Connections): all routes * Secondary Peers (ESnet, NASA Internet): only routes of primary peers. Customers cannot talk to other customers through vBNS - policy mapping/policy routing using src IP address will suffer from severe performance penalty for OC-x. - QoS issues * No router can provide full QoS at OC-3/DS-3. -> Bandwidth reservation per connection. * WRED (Weighted Random Early Discard): ** being studied at IETF. ** MCI pushes Cisco to implement WRED on OC-12. Only supported on RSP2, not on VIP2 (June, 1997). * WRL (Weighted Rate Limit): ** uses 3-bit "precedence" field in IPv4 header. ** IP CIR (Committed Information Rate) * Mixture of UBR and VBR on OC-12 degrades the total throughput. * Some ATM switches degrades the throughput as VCs/VPs increase. * Mixture of ATM-native applications and IP applications make the bandwidth management extremely complicated. - MCI participated in Cisco RSVP beta test. * 96/08-96/11 beta test of Cisco IOS-11.2 * final testing at SC96 with NCSA, CTC, RES. * next generation of implementation: Feb., 1997 * IP level RSVP is better than ATM level QoS, considering IP over SONET configuration. * vBNS issues ** not so much congestion for testing RSVP. ** no scheduling algorithms to keep up with OC-3/OC-12 ** no mapping between RSVP and Q.2931 (ATM QoS) - Multicast * vBNS adopts PIM. * application ** web cache multicast ** OC3MON active flow updates ** Videoconference ** Mbone session * in 1997 ** underlying support for RSVP ** coast-to-coast Mbone transit ** CANARIE PIM -- supposed to be up this afternoon ** expansion of PIM cloud ** ATM-level multicast ** multicast-specific measurements on vBNS via OC3MON - Network Management * NOC: 24 hours a day, 7 days a week. 24 members for both vBNS and commercial ISP. * SNMP based Network Management ** HP OpenView, FOREview, Notel... * NEXPERT (MCI internal): day-to-day monitoring, inference, capacity testing, conformance testing, simple reachability * OC3MON ** low-cost, non-intrusive(passive) ** now developing OC12MON. 2. APAN issues (questions and comments from MCI) - XP configuration: will work. - why are there two ATM switches at an XP? * ATM layer overhead is considerable on international links. - addressing format of MCI-ATM: NSAP - ATM Switch: APAN may select StrataCom. - Fast Ether switch: smaller MTU may affect performance. - Platform for IP servers: your mileage may vary. "Buy the same thing (up and running) to do the same thing." - APAN will need automated configuration for PVC/PVP setup. - ABR: MCI stops believing the vendors' schedule. - APAN-vBNS MBone connection: interesting. 3. Lunch with NSF Mr. Goldstein, Mr. Wilder, Mr. Kikuchi, Mr. Youn and Mr. Konish - Tender will be announced in April, - High performance applications with names of principal researchers should be included in our proposal, - Supporting program for developing countries is not required. - MCI's vBNS team is separated from the commercial team. The exception is Dr. Cerf. --------------- ARGONNE MEETING --------------- 1. Activities around Chicago Area - goals: high level, tele-immersion research testbeds - research themes: * tele-immersion networking * human/computer interaction * operating system / system software -- for real time * graphics & visualization algorithms * VR display & simulation Hardware * VR/Video/Audio Integration - networking research activities * UIC, MREN, vBNS * Internet II -- consortium of universities * Next generation internet -- federal government program * STAR TAP: collaborative virtual prototyping environment supporting global product development using ATM. (NCSA) * IWAY physical connectivity Iway, AAINET, ATDNET, MAGIC, ESNET, NASA NREN, BAGNET, CASA, vBNS -- ATM based ** The way people work together in the future. * GIBN: G7 global interoperability for broadband networks -- to solve financial problem only - MREN: Metropolitan Research & Education Network * BGP4 peering: every net is an autonomous system * Full PVP mesh - Star Testbed(or Transit) Access Point * NSF high performance network connections program * 100+ top us research univ. ** should have "driving application", campus wide network ** should write a proposal to NSF * OC12 to SCCs (Super Computer Centers) * DS3 - OC3 to universities and work sites. - vBNS AUP * vAI: vBNS Authorized Institution ** vAIs may use the vBNS to exchange traffic among themselves and with vPIs. * vPI: vBNS Partner Institution (APAN may be a vPI) ** vPIs may use the vBNS to exchange traffic with vAIs but not with each other. * Neither may use the vbns to exchange traffic with non-approved sites * BGP community will be used for AUP * Partners may be non-US institution * Good faith basis, no monitoring function - Internet II * 150 research univ. * High performance application * Flagship is QoS * Aggregation of sites into GigaPop - Next Generation Internet (NGI) * initiated by president * 10 sites at 1000 times current internet speed --> OC3 * 100 sites at 100 times current internet speed --> DS3 * vBNS/Internet2/NGI -- which to connect? ** vBNS initially worked for the connections. - STAR TAP * Goal: high performance distributed applications on a global scale * Objectives: IP and ATM provide new network services rapidly RSVP & ATM QoS IPv6 BGP communities & policy based routing SVC IP Multicasts * Feature: Layer 2 services provided by AADS network engineering, application, documentation and publication SUPPORT ** Network Support: Link establishment Routing configuration PVC/PVP configuration Problem isolation and resolution Statistics -- ATM & IP * Chicago NAP / STAR TAP / MREN engineer ** Same switch. Small(8 persons) and the same maintenance team ** Participation in Internet2, MREN and vBNS engineering 2. Technical Details of TAP - uses ASX-1000 - the biggest problem: policy routing - benefit of BGP4: communities -- attaching policy info. to routes - FISH problem should be solved: ** only the "best" link will be selected, even if there are several links with different policy/preference. ** This is caused by DESTINATION BASED ROUTING. (same diagram as the MCI presentation) A ----D---- \ / \ C G / \ / B ----F---- * A can transit only D to reach G. * B can transit only F to reach G. * C can transit both D and F to reach G. ** Cisco has the solution and developing NEW HARDWARE. For the time being let them go. 3. AADS (Ameritech Advanced Data Service) - Started with about $30 million. - AADS and customer relation: sometimes customer helps AADS to make technological decision. - Test Lab has Globeview/StrataCom/Cascade. - Issues: ** How network configuration can change themselves? ** Globeview shall be replaced. AADS has a strong relation to Cisco( StrataCom? ) 4. CANARIE - National Test Net: much like vBNS. - CA*NET II: similar to internet II. - Objective * Partnership between industry, carriers, regional advanced networks and R&E community * catalyze the building od a sustainable Virtual R&E network - Technical Architecture * Buy vs. Build: Buy. * Regional vs. National: Regional. * IP QoS vs. ATM QoS: IP will be the common bearer service. * Core vs. Edge: network intelligence should go to edge. - CA*net II model BAC Network -- Teleglobe -- vBNS |- AT&T network -- vBNS |- GigaPop -- R&E networks |- Regional networks 5. APAN issues Q: Are you planning to charge end-users? - No. C: RSVP capable routers may have charge function. Q: Who is the contact of medical applications? - Dr. Mizushima. Q: Who / How to divide and handle reserved 30Mbit/s? - not decided yet. Q: Clarification: Who pays transpacific link? Who can use the link? - JST pays it, and APAN members can use it. C: Assignment procedure should be clean. Link tends to be filled out. C: Application people do not know the bandwidth they want, but do know how much bandwidth APAN has. The request form should be simple. Q: What cache software? - Squid, for the time being. C: Issue: java caching C: Large RTT issue on TCP performance. Large window size, multiple TCP connections,... Same technique can be used on satellite links as well. 6. Discussion - ATM overhead: packet loss can be as much as 30% - IP/ATM or IP/SONET? SONET is good for edge network. To maintain nation-wide network, to make full-mesh configuration, it becomes headache. - connectivity to vBNS with SONET/SDH: unknown. Not trivial. - NSF tender will be announced in April. Maybe Mr. Butler can help APAN with that matter in the US side. 7. Demonstrations at ANL - Iway * to break the barrier for collaboration, time and distance. * to create new things with different communities: high performance computing, high speed network and application. - ATM switch * Ameritech GV2000 * 3M fiber cross-connect (jumper) - CAVE * Virtual Reality with 3-D glasses 8. MREN Executive Meeting - www.mren.org - history started at UIC in dec. 1996 will disclose the plan in April workshop - development activities * developed implementation plan closely tied to ANL MORPHNET document (blue one, open on web). * follow through with AADS to identify and prioritize list of activities. * work closely with AADS engineering to investigate new technologies. - MORPHNET: production level bar layer 1: SONET layer 2: ATM layer 3: IP,IPv6 layer 4: RTP, RSVP,.. layer 5: Applications - What is in it for application? * run code over production nets - keep the edge! * will not have sacrifice future advanced technology for today's production network * can use advanced network technology, QoS, active network control * use un-mined bandwidth - key concepts * production vs R&D * integrated services model * admission control * monitoring and management * experimental network and testbed - Similar efforts * CANARIE * CA*net II * APAN - Discussion * Internet2 - engineering meeting in San-Diego will define "what the GigaPop is." See www.internet2.edu * See vBNS web page for the Applications. * MREN will develop network tools, ex. traceroute over ATM. * Contact: Linda Winkler. * vBNS fee -- MCI wants to charges access fee to vBNS. It is an aggressive price, research network couldn't accept. -> New pricing is now being studied. C: Mbone is nice, but it is a toy. -------------- OTTAWA MEETING -------------- Owing to the double accidents of Northwest's flight, 5 members (Prof. Goto, Mr. Konishi, Mr. Hor, Mr. Tay and Fujinaga) could not get Ottawa as scheduled. 1. CANARIE - Non-profit corporation - A small staff (20 members) but has a membership of about 140 organizations. - Daily operation is left to telecoms. - NTN: Test network CA*net: production research network CA*net2: NTN -> research network - Infrastructure Project priorities * Key focus on Carrier partnership and emphasis on deploying infrastructure to support SVC/PVP virtual cloud, QoS program * the highest priority : ARDNOC and infrastructure deployment(to get off the ground 2nd Q 1997) cf. ARDNOC(Advanced Research and Development Network Operated by Canarie) ** carrier/research activities ** responsible for - coordination CA*Net2 routing and management(US, International) - coordination of routing arbiter, IPv6, pim Mbone, 6Bone, global caching, Internet Telephony etc. - Configuration is ALWAYS changing. Now moves from 3 DS-3 links to dual OC-3 links. - CANARIE pays for TAP link of DS-3. - Differences from vBNS * vBNS allows IP connection by apporoval, rather than ATM connection while CANARIE allows all connections by approval. * vBNS-like AUP, but mostly acceptable if the traffic is generated by R&E * Closely related to commercial industries - More than 20 universities peers each other in full-mesh BGP4 configuration using PVP and PNNI SVC through "GigaPops". - ATM migration: phase 1: IISP phase 2: PNNI phase 3: PNNI2 - IP Broadcast: ADSL to the home (more than 2000 subscribers). - ATM switch: StrataCom (because of its big buffer) - Key to success: massive collaboration of industry, education and government. - Keep it in mind "How to live with interesting things to come." - Migration from NTN to CA*net2 was with no trouble. (from June 1996 to October 1st) - The very first stage was painful, nearly 60% of IP packet loss. 2. CRC/OCRInet - Ottawa Carleton Research Institute - industry, government, univ. - ATM network - isolated - education - research - local subscribers - Highway Infrastructure Program - Tourism project - will be delivered over ISDN - Multilingual(French/English) - Current Status - dual DS-3 backbone - single DS-3 branch - to other networks - ATM T1 satellite - E3 -> Europe - broadband service(5 channels) - Multimedia News Delivery - cooperation <-> local newspaper - Dynamic Arts - Full Motion Video(30fps) - commercial network(not exchanging) - trans pacific network analysis(plan) 3. BADLAB - Broadband Applications and Demonstration Laboratory - ATM high speed network - Application development - Education - Medical - artificial heart demonstration - Scientific Applications - Engineering - Government service - G7 project - satellite - TPC4... - demonstration - MPEG2 HDTV (G7) - operating room - evaluation room 4. Newbridge - has 45% share in ATM switches, reported by International Data Corporation(IDC) - TDM(time division multiplexing) - WAN/LAN interface - OC-12/STM-4 - OC-3/STM-1 - T3/E3 - T1/E1 - FDDI - TokenRing - is used for Bank System: - automatic tellermachines - voice - fax - designed by CTA(Communications Technology Associations, Inc.) - Frame Relay & ATM Internetworking - FRATM(Frame Relay to ATM) - broadband wireless network - LMCS(Local Multipoint Communication Systems) : Canada - LMDS(Local Multipoint Distribution Service) : USA - licensed by Industry Canada or FCC - accesTV(bi-directional TV) - military - trooper II chip for VME bus ---------------- MONTREAL MEETING ---------------- 1. Teleglobe - Overview of Teleglobe Teleglobe Canada Teleglobe Media Enterprises Teleglobe International Teleglobe USA/UK/Germany Teleglobe World Mobility (world wide paging service) * 240+ countries, 147 direct link, * competitors Canada: Stentor, AT&T Canada, Sprint USA: RBOC, Resellers, Overseas carriers * Cable section is active. consortium of cable companies: see www.vision.com - Overview of APAN Q: Who pays what? 2. Teleglobe INOC - Switches: Nortel DMS-300/100 - Cross connect: Newbridge/Marconi - Operation of X.25, FR, ATM, Internet services - 24H/7d - ITU TMN based OAM&P system (first implementation in the world) 3. Globeinternet - Global transit service and access to North American Internet - Services: IPv4, DNS, Network Time, News and MBone - Future plan: RSVP, 6Bone - Videotron: Gateway to Internet cable TV. - Nodal Architecture (access point) * Cisco LS1010 ATM switch * Fast ether hub * Cisco 7513 Router * ULTRA SPARC for IPv4 servers 4. ATM - Already connected today: * CANARIE NTN/BELL (45Mbps) * Japan (Satellite 45Mbps, KDD/NTT) * MCI/vBNS (via CANARIE and TAP) * JAMES (ATM Trials in Europe) - Teleglobe can provide Asian networks full global reach. - Teleglobe provides T3 satellite link to Japan for GIBN project. * Will CRL connect the link to SINET/IMnet? - Teleglobe is ready to work with APAN. Cost sharing is the key issue. ------ ISSUES ------ - Network Design * An ATM switch shall be installed in each XP. * PVC/PVP request form shall be simple. * Automated PVC/PVP assignment procedure * Will Cisco add enough buffer memory to StrataCom BPX? * Will the US partner adopt the same ATM SW? - NSF proposal * High performance applications and their researchers - Cost sharing ----------------------------- APPENDIX 1: APAN TRIP MEMBERS ----------------------------- Shigeki GOTO Waseda Univ., goto@cfi.waseda.ac.jp Chan-Hyun YOUN Korea Telecom, chyoun@hudson.kotel.co.kr Kazunori KONISHI KDD R&D Labs., konishi@lab.kdd.co.jp Shunichi KIKUCHI JST, skikuchi@tokyo.jst-c.go.jp Yasuichi KITAMURA CRL/MPT, kita@crl.go.jp Naohiko NAKAJIMA JST, nakajima@tokyo.jst-c.go.jp Shuichi TASHIRO ETL/MITI, tashiro@etl.go.jp Masahiko FUJINAGA KDD R&D Labs., fuji@lab.kdd.co.jp Sang-Eon KIM Korea Telecom, sekim@noha.kotel.co.kr Jinhyoun YOUN Korea Telecom, jyoun@waltz.kotel.co.kr Tommy HOR Tze Man Computer Centre/NUS, ccethor@nus.sg TAY Sen Yong ISS/NUS, senyong@iss.nus.sg ############################################################ DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT ############################################################