Re: [RRG] Re: Comparing APT & Ivip - new business models

[Robin Whittle <rw@firstpr.com.au>]

Hi Michael,

I think our recent discussion highlights a major difference between
the business models in which you view APT and potentially Ivip being
adopted and how I anticipated Ivip being adopted.

I think you see the end-user network retaining their current
relationship with their ISP, and the ISP deploying the scalable
routing system (APT or Ivip) for the benefits it brings to the ISP.

I see the end-user network adopting Ivip by creating new business
relationships with entities which do not currently exist, in part so
that the end-user network will no longer be so dependent on any one
ISP.  I believe ISPs will adopt Ivip to attract new business and/or
to retain existing customers who wish to adopt Ivip.

I think you are intending APT to appeal to ISPs, while I am
intending Ivip to appeal to end-user networks.  That said, since
ISPs are generally keen to serve the needs of end-user networks
(existing customers and potential new customers) it may well be ISPs
who take a lead role in implementing Ivip and in trying to inform
end-user networks of its value.


> Below, you describe your doubts about how a single ISP could deploy APT
> unilaterally, without the involvement of their customers. Allowing for
> this, and giving ISPs an incentive to do so, is perhaps *the* primary
> goal of our incremental deployment scheme. As I mentioned before, we
> should have a new document describing the updated details sometime in
> the next few months. But, to summarize, you can think of a single ISP
> deploying APT as similar (in concept) to a single ISP deploying MPLS, or
> some other internal efficiency improvement. 
             --------

> The difference is, APT allows for a potential increase in benefits with 
> every other ISP that deploys it.

I understand this as "If an ISP adopts APT, it will gain some
internal efficiency benefits.  If other ISPs adopt it as well -
presumably in the same APT island as this ISP - the efficiency
benefits for this ISP will be increased."


> This relates to another point below regarding non-addressability of
> internal routers in an APT ISP. Put simply, since all traffic through an
> APT ISP is encapsulated (even "legacy" traffic), only the border routers
> of that ISP can possibly be addressed from outside that ISP.

I could imagine that if the Internet consisted solely of
APT-equipped ISPs and edge networks, all of which use APT, then
there would be a true separation of core and edge, with every edge
to edge packet going through an ITR and ETR, with encapsulation.

However, what about two edge networks E1 and E2 which are both
solely using ISP1, and therefore the same ITR(s) ETR(s) and DM(s)?
Do you encapsulate and tunnel the packet to the local ETR?  I am not
suggesting you shouldn't - but I wondered in this case whether the
APT system would somehow avoid encapsulation when there is no need
to send the packet outside the one ISP.

I can't understand your description in the transitional phase - when
only some edge networks are using APT.  I will refer to non-upgraded
networks.  I don't like the term "legacy" for various reasons,
including it seems to place a negative value judgement on networks
or technologies which simply sit there and mind their own business,
not being swept along in the latest fad, upgrades or whatever.
There is a lot to be said for leaving a working IT system alone!

Let's say end user network E3 is non-upgraded.  It is a conventional
PI network with a single prefix advertised in BGP.

How could a host in network E1 (upgraded to APT) send a packet to a
host in E3 according to this?

   all traffic through an APT ISP is encapsulated (even "legacy"
   traffic),

There is nothing at E3 to decapsulate such a packet.  The only way
this could work is if the ITR(s) and DM(s) in E1's one or more ISPs
do not alter the packet at all, but forward it to the border router
which forwards it via the other DFZ routers to E3's border router(s).

Likewise, when a host in E3 sends packets to a host in E1, these
packets arrive in their raw state, after travelling through the DFZ,
and the border router of the one or more ISPs in E1's APT island
which advertise a prefix which encompasses E1's prefix.  (Actually,
to the one or more border routers of ISPs where those border routers
advertise the most specific prefix in the DFZ which encompasses the
address of the host in E1.)

I understand that those border routers just mentioned operate
somewhat like LISP PTRs or Ivip OITRDs (Open ITRs in the DFZ) in
that they attract packets from non-upgraded networks, hopefully
fairly close to the non-upgraded networks in order to ensure the
shortest paths no matter where E1's current ETR is located.  Then,
they encapsulate the packet and tunnel it to whatever ETR that
border router (which is an APT ITR, working with a DM) decides is
the best one to use, according to the current mapping of E1's EID
prefix, and the reachability of the multiple ETRs specified in that
mapping, the priorities in the mapping etc.

So as long as there are non-upgraded networks, there will always be
packets from non-upgraded edge networks entering APT ISP networks
from the DFZ in their raw, non-encapsulated, form.  Likewise, any
APT ISP which has a packet to send to a non-upgraded edge network
will be sending it in a raw form across the DFZ.


> Regarding our discussion on economics and policy: I think your
> description is backwards. Nothing will be deployed on the Internet as it
> exists today unless someone has an economic incentive to do so. 

I agree with your second sentence, but not the first.

> So, we must constrain our technical solutions to meet the needs of 
> real-world economics, not vice versa.

Yes, but we are not restricted to current business relationships,
since we can and should invent new business entities, new kinds of
relationships etc. if this is important to the working of the new
scheme.

If you think you can solve the routing scaling problem without
altering any existing business relationships, or without creating
any new ones, new kinds of business etc. that is fine - your plan is
arguably simpler and perhaps therefore superior to any plan such as
mine which involves new business entities and relationships.  I
can't imagine solving the problem in the best possible way, or of
making a plan which is attractive to end-users, ISPs and whoever is
to run the new mechanisms proposed, without new business entities
and relationships.


> When I say that your solution is an economic one, I mean that it depends
> upon the success of a number of business models and economic structures
> that do not exist in the Internet today. 

Yes.


> If no one thinks they can make
> a profit off of being a RUAS, MRD, or so on, and you cannot convince
> them otherwise, then Ivip won't work. Even if you do convince them
> otherwise, if no one is willing to pay for the services of a RUAS, MRD,
> or so on, Ivip also won't work.

I agree.


> With APT, we have focused on providing a solution that can be deployed
> *within the economic constraints that exist on the Internet today*. That
> is, if an ISP feels that deploying APT is a good investment in their
> business, they can do so without any new business partners, contracts,
> payment plans, or so on.

If you can do it on this basis, and have APT widely adopted simply
by most or all ISPs deciding it to their advantage, without needing
to consult with their customers, then that would be a far simpler
business arrangement than what I anticipate with Ivip - so it would
be a superior solution if it produced the same benefits.

Ivip was prompted by the need to solve the routing scalability
problem, but some of the benefits which I expect to result,
including it being the basis of a new form of global mobility, seem
to be unrelated to routing scalability.  However, the routing
scalability problem is the sole reason why we can't do single IP
address global mobility right now.  If there was no scaling problem,
an end-user could could advertise any prefix at any ISP and have
this propagate through the whole DFZ in a few seconds, without
anyone feeling their router costs were being unfairly burdened by
billions of such rapidly changing prefixes.


I think it is the end-user networks which need to make the decision
and investment to adopt the scalable routing solution - or at least
any core-edge separation solution such as LISP, APT, Ivip or TRRP.
Therefore, I intend Ivip to be of interest primarily to end-user
networks for reasons of being able to cost-effectively gain access
to portable, multihomable "Scalable PI" (SPI) space compared to any
  alternative.

The scaling problem directly affects any ISP which runs routers in
the DFZ.  The costs for each ISP's DFZ routers are imposed by any
network, but mainly by the more numerous end-user networks rather
than ISP networks.

The end-users in general - PI and PA - don't care about routing
scaling, except perhaps in a very broad sense in that it puts up the
costs of all ISPs and therefore generally increases their cost of
connecting to the Net.


At the same time, the routing scaling problem is the primary reason
why millions of end-users which want portable, multihomable, address
space can't get it.  If there was no routing scaling problem, then
it would be fine to advertise hundreds of millions of prefixes in
the DFZ, including of any length, such as to /32 for IPv4 - and
change them at any frequency.

Because there is a scaling problem, ISPs and RIRs greatly resist
handing out smallish blocks of space to anyone (end-users generally,
since ISPs only want larger blocks).  Likewise they strongly resist
the advertisement of smaller blocks, such as with the widely
respected, though informal, /24 limit on length of prefix accepted
by most DFZ routers (except perhaps for special, specifically
allowed, longer prefixes).

This means that millions of businesses, schools etc. can't get
portable, multihomable space - so if they want reliable
connectivity, they need to use PA space, which is not portable and
locks them into the one ISP, due to the costs and disruption of
renumbering to the PA space of another ISP.


So while end-user networks are strongly impacted by the routing
scaling problem, they are not going to spend money or make any
change out of the goodness of their hearts to try to solve the
problem.  The only way large numbers of them will adopt a new
technology which happens to solve the routing scaling problem is if
it gives them direct benefits in a short time-frame, such as days or
months.

I am certain that it is end-user networks which make the decision
about whether or not to adopt APT or Ivip.  I wrote in recent
messages about how I can't imagine APT being adopted by an ISP for
an edge network's address space without the consent of that edge
network.  You haven't shown what is wrong with my attempt to
understand how this could be done - I couldn't see a way.


There are three ways an end-use network could adopt Ivip.

Firstly, if they currently have a PI prefix, they could keep that PI
prefix and manage it with Ivip, turning into a MAB, or making it
part of a larger MAB, as I described:

  Renumbering once might be OK when converting to Scalable PI (SPI
  space) http://psg.com/lists/rrg/2008/msg02355.html  2008-08-26

Secondly, if they had PI space and decided to relinquish it, or they
had PA space and likewise relinquished it, they would gain some new
address space from a MAB operating company, by renting it on an
ongoing basis - as also described in the just-mentioned message.

Thirdly, new networks would establish themselves on space rented
from a MAB operating company.


The primary reason for an established PI end-user network converting
its PI prefix to Ivip would be that it could split the space very
finely, not restricted by the BGP /24 limit, down to single IPv4
addresses if it liked, and map them to any ETR in the world.  So it
would be able to make much greater use of the space than if it
retained the current BGP arrangements.  Converting the prefix to an
Ivip MAB may be the only option, or a cheaper option, than gaining
enough PI space to multihome a larger number of geographically
dispersed sites each via two or more local ISPs, using conventional
BGP techniques which limit the size of the prefixes to 256, 512
addresses etc.

There could be various reasons why an established PI network would
want to give up its existing space and instead rent space from a MAB
operating company.  For instance, the end-user network would no
longer need to be involved with an RIR, which could be quite a
financial and administrative saving.  It would no longer need to run
BGP routers.  It may be able to meet its addressing needs with a
smaller amount of space than it currently has, due to the ability of
Ivip SPI space to be split arbitrarily finely, including into any
number of contiguous IP addresses, not just power of 2 prefixes.  In
particular, as just mentioned, the space could be split into smaller
blocks than 256 addresses for the purpose of multihoming many more
sites in a smaller total range of addresses.

Since IP address space will increasingly cost money (or
alternatively be a source of profit if sold or hired out to someone
else), no matter what its form (as a market develops for space and
as demand grows), any arrangement which enables an end-user network
to meet its needs with less address space is likely to lower the
costs for that end-user network.

Ivip also enables real-time load balancing of incoming traffic over
multihomed links, at the cost of the mapping updates, provided that
the incoming traffic is spread over multiple IP addresses which can
be in separate micronets.  (If the traffic is for a single server,
it is easy to give the single server multiple IP addresses and put
each address in a different micronet, so it can be mapped to a
different ETR.)  This is arguably a finer-grained and more
dynamically adjustable form of load sharing than is possible with
current BGP arrangements, so it may translate into the end-user
network being able to save on the costs of its ISP links.  With
Ivip's approach to TE, the end-user network may be able to handle
incoming traffic levels adequately on links which have less capacity
and are therefore less expensive than the faster links which would
be required if TE was done only with existing BGP techniques.


I think there will be a strong demand for SPI address space, and
that new or existing companies will go into the business of getting
prefixes, turning them into MABs and renting out the space as SPI
space - User Address Blocks - which can be split into as many
micronets as each end-user likes.  Also, these MAB operating
companies would provide OITRDs for their SPI customers.  Likewise,
the MAB company would pay a new RUAS company to handle the mapping
changes.  The RUAS would charge the MAB company for this and the MAB
company would charge its customers per mapping update, or on a
flat-fee basis with some monthly limit.

I am quite happily conjuring up new business entities and business
relationships because I think this is the best way to make a routing
scaling solution which is attractive to end-users and to the
companies who will need to build the scaling solution's infrastructure.

While I am proposing this as an IETF standardised system, with
multiple operators of MABs, multiple RUASes etc. with the RUASes
forming a consortium to run the Launch server system and the initial
levels of the Replicator system, the whole thing could be done
without any IETF standards right now (with a lot of work, of
course).  (This would have to be on an encapsulation basis - the
Forwarding methods now in Ivip are only useful once most or all DFZ
routers have been upgraded, which is impossible for a single company
to ensure.)

A company could develop software, run servers around the world to be
OITRDs, develop a mapping system, and create a standard protocol for
ETRs to work with their ITRs.  Then they could make the ETR and ITR
software for COTS servers available for free to ISPs and start
renting out MAB prefixes as SPI User Address Blocks for customers.

This could be a little difficult to get off the ground, since as
described it would be for non-mobile networks, and it might be
tricky to get ISPs to invest in setting up ETRs, administering them
etc.  Still, it might work out OK, especially if the scheme had a
backing of a big company whose forays into new fields had a history
of being successful.

Another approach to introducing an Ivip-like system would be to do
as just described, but not focus so much on non-mobile networks,
with their needs for ISPs to install ETRs and ideally ITRs.  The
scheme would be to run TTRs (Translating Tunnel Routers) all around
the world and to sell globally portable IP address space which is
accessed from any care of address whatsoever, including behind NAT,
to customers who want one or more stable IP addresses for their
laptop, cellphone etc.  This would give session continuity when they
change from one access network to another - with consequent change
of the care-of address:

  http://www.firstpr.com.au/ip/ivip/TTR-Mobility.pdf

In this model, a single company would do everything: run the mapping
system, Replicator system etc. (No need for separate RUASes.)  It
would have all the MABs in this system, and would run the OITRDs and
the TTRs.  This could be a profitable quite rapidly, I think.

Multiple such businesses could co-exist, each with its own mapping
system, TTRs etc. - including if the various technical standards
were incompatible.

In all these cases it is the end-user network themselves which needs
to make the move to adopt Ivip.  I can't imagine how APT or LISP
could be any different.  I can't imagine how any of these schemes
could be applied to an end-user's address space without the end-user
network administrator needing to consent to this, since the
integrity of their network's connection to the net, the costs
involved, the technologies, the capabilities etc. would be
completely different after the adoption.


> I believe that, though the economic solutions proposed in Ivip may be
> profitable or useful to someone, it is nearly impossible they will be
> profitable or useful to *everyone*. 

I agree.  If an ISP is happy with its locked-in customer bases of
end-user networks on PA space, then Ivip is a threat to that happy
(for the ISP, not for the end-user networks) arrangement.  Ivip will
provide a low-cost, flexible, method of gaining a new, scalable,
form of PI space which suits the needs of many of these customers,
in particular by giving the customer the opportunity to move to
another ISP and to do multihoming with two or more ISPs.

Such an ISP will lose some of these customers.

If such an ISP doesn't implement Ivip ETRs and/or try to convert its
existing customers to Ivip so as to retain their business, then the
ISP will definitely lose business compared to ISPs which adopt and
promote Ivip.

Ultimately it is to the benefit of all ISPs - and therefore to all
end-user networks - that a scalable routing solution such as Ivip is
adopted.  However, some ISPs could lose business, as just described,
if they either don't adopt Ivip and/or their service and costs are
not competitive with those of other ISPs.  This is as it should be.


> We would be better off incorporating these solutions into other
> systems that do not *depend* upon them.
> 
> One example is charging for BGP updates (see other thread), another is
> charging edge networks to update the TE info in their mappings under
> APT. This is certainly possible, since we haven't specified any
> automated way for an edge network to update their TE info, each provider
> can choose for itself if this is something they should charge their
> customers for. But they would never be *required* to do so in order for
> APT to function efficiently.

But what if some customer decides it wants to change its mapping
every hour, or every 10 minutes?  If the APT mapping distribution
system is working nicely, the changes might propagate across the Net
in a few minutes, to 10 or 30 minutes or whatever.

Suppose an end-user network likes this and finds they can use the
APT system to dynamically control their load balancing by changing
their mapping like this 24 hours a day.

Then, I am sure, other ISPs in the APT-Island will develop a keen
interest in charging any ISP in the island which continually burdens
their networks with this flood of updates, which you as the APT
designers do not currently anticipate will occur.

The Ivip architecture anticipates and encourages end-user networks
to change the mapping as often as they desire, according to the cost
per update and the benefits they gain.  Since most of the mapping
system is run as a profitable series of businesses, financed by the
charge per update, then the update activity will always be
constrained by the real costs of fanning the update out across the
globe.  At the same time, the infrastructure will be built to carry
these updates efficiently, so more can be handled at a lower cost.
Different RUASes would compete for the lowest cost for mapping
updates, but they need to work together to run the common parts of
the system: the Launch servers and some or many of the Replicator
servers.

With Ivip, there will be a greatly reduced threat of a tragedy of
the commons, compared to the BGP situation at present and as there
would be with APT - the problem of some free-rider pumping
"unreasonable" numbers of updates into the system.

This absence of unfair costs is not entirely true of Ivip, as
currently planned - the ISPs who run full database query servers
will face costs per update and per micronet, in terms of mapping
update traffic, storage and CPU time.  They will probably also be
running the last few levels of Replicators:

  http://tools.ietf.org/html/draft-whittle-ivip-db-fast-push-01

If there is some bunch of end-user based in Asia who are pumping out
lots of updates, either for load sharing TE or due to some desire to
keep using very nearby TTRs when they are highly mobile between
different cities, or different access networks in the one city, then
why should an ISP in Zanzibar or Chile, which hardly ever sends
packets to these end-user networks, accept all these mapping
changes, use up memory and CPU cycles in their full database query
servers etc.?

There is further work to do on this, so I don't claim that all
aspects of the Ivip business model ensure no-one has costs which
they are not paid for.  However, Ivip is a very different approach -
more expandable and scalable for technical and business reasons -
than BGP or APT.

 - Robin


--
to unsubscribe send a message to rrg-request@psg.com with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg