HomeMy WebLinkAboutStaff Report 14742
City of Palo Alto (ID # 14742)
Utilities Advisory Commission Staff Report
Meeting Date: 9/14/2022 Report Type: VII. NEW BUSINESS
City of Palo Alto Page 1
Title: Colleagues Memo From A.C. Johnston, Phil Metz, and Loren Smith:
Implementation of a City-Owned FTTP network and City-Owned Internet
Service Provider
From: Director of Utilities
Lead Department: Utilities
Discussion
The purpose of this memorandum is to provide the results of the UAC sub-committee for FTTP
Initiative investigation and work with city staff and 3rd party Magellan Advisors on the
proposed implementation of a City-Owned FTTP network and a City-Owned Internet Service
Provider (ISP).
Attachments:
• Attachment A: Colleagues Memo
6
Packet Pg. 78
MEMORANDUM
Date:
To:
From:
14 September 2022
Utilities Advisory Commission, City of Palo Alto
Utilities Advisory Commission, Sub-Committee on FTTP Initiative
Vice Chair A.C Johnston, Commissioners Phil Metz and Loren M. Smith
Subject:Implementation of a City-Owned FTTP network and City-Owned Internet
Service Provider
The purpose of this memorandum is to provide the results of the UAC sub-committee for FTTP
Initiative investigation and work with City Staff and 3rd party Magellan Advisors on the proposed
implementation of a City-Owned FTTP network and a City-Owned Internet Service Provider
(ISP).
Based upon our research, a complete review of the business and financial models, as well as
the results of the City of Palo Alto survey, we recommend that the UAC recommend to the Palo
Alto City Council that the Council authorize CPAU to proceed forward with plans to offer fiber-
based broadband services through a City-Owned FTTP network and a City-Owned Internet
Service Provider to all Palo Alto residents. We also provide some specific recommendations to
address the risks inherent in establishing a city owned ISP.
A Brief History of FTTP in the City of Palo Alto
For more than 20 years, the City of Palo Alto has investigated, engineered, built and operated a
dark fiber network. Originally conceived in the mid-1990s, the City’s initial telecommunications
strategy was to build a dark fiber ring around Palo Alto that would be capable of supporting
multiple network developers and/or service providers with significant growth potential. The first
phase of the fiber network construction occurred in 1996 - 1997 and consisted of 33 route miles
with 144 or more strands of single-mode fiber along most routes. After the “dot com bubble”
burst in 2001, the City’s efforts were more subdued, but since the late 1990s, the fiber backbone
has been expanded to approximately 49 route miles of mostly 144 or 288 count single-mode
fiber.
Page 1
6.a
Packet Pg. 79
At
t
a
c
h
m
e
n
t
:
A
t
t
a
c
h
m
e
n
t
A
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
(
1
4
7
4
2
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
)
In 2000, the City began to license “dark fiber” for commercial purposes. Dark fiber is unused
fiber through which no light is transmitted, i.e., installed fiber optic cable that is not carrying a
signal. Today, the City currently licenses dark fiber connections to 99 commercial and City
customers, including the following City accounts: IT Infrastructure Services, Utilities
Substations, Utilities Engineering, Public Works, Water Quality Control Plant and Community
Services (Art Center) yielding a total number of dark fiber service connections of 162 (serving
commercial customers and the City). Over that same period, the Fiber reserve balance at the
end of FY 2022 is approximately $34 million (Actuals anticipated end of September 2022).
In this same period, going on 15 years, the City has studied, planned and worked to develop a
business case to build a citywide Fiber-to-the-Premises (“FTTP”) network to serve homes and
business (Reference: “History of the City of Palo Alto Dark Fiber Optic Backbone Network”, Ver
6.0, 7 August 2019). On June 24, 2019, the Utilities Advisory Commission (UAC) was directed
by City Council to assume the primary advisory role and serve as the public input forum for fiber
and wireless expansion initiatives by the City of Palo Alto. Since then, the UAC has devoted
significant attention to how this underutilized asset can be used for the betterment of the City of
Palo Alto population. On June 24, 2019, the UAC recommended and Council agreed to leverage
the existing dark fiber network into benefiting its infrastructure, e.g., emergency services, city
owned buildings and our pending AMI network. On June 1, 2020, the UAC recommended and
Council agreed to engage Magellan Advisors (Ref.: https://www.magellanbroadband.com/) and
on May 24, 2021, the UAC recommended and Council agreed to accelerate the consultant’s
work, including to:
1) Design and engineer the expansion of the City of Palo Alto’s dark fiber network;
2) Engineer the incorporation of an AMI network into the network;
3) Engineer the incorporation of a FTTP solution into the network; and
4) Provide a business case and associated financial model for the development of a FTTP
ISP service offering by the City of Palo Alto.
The once-in-a-generation event that was the COVID-19 pandemic (declared a pandemic on
March 11, 2020, by the World Health Organization) highlighted the many dependencies our
population may have taken for granted, namely:
1) Our food supply;
2) Other necessary supplies, including bathroom tissue;
3) The resiliency of small businesses; and
4) The need for Broadband communication.
Indeed, our experience during the initial months of the pandemic shed light on the fragile nature
of our supply chain and our services. Not to be left out, as our offices were shuttered, and our
population migrated to work-from-home (WFH) and our children attended school-from-home
(SFH), the stresses on our broadband infrastructure eroded our ability to function routinely. In a
matter of days, the necessity for broadband services moved from a convenience to an “essential
Page 2
6.a
Packet Pg. 80
At
t
a
c
h
m
e
n
t
:
A
t
t
a
c
h
m
e
n
t
A
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
(
1
4
7
4
2
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
)
service/public good,” similar to electricity, water and sewer, and became a non-negotiable
requirement of our community. As if by luck, our City had already engaged Magellan Advisors,
and a suitable, appropriate and deliberate response was already in the making.
Why does the City of Palo Alto need to build a city-owned FTTP network?
Broadband service to the City of Palo Alto’s population is currently dominated by two large
companies: AT&T and Comcast/Xfinity. Recent survey results by Magellan Advisors reveals a
population concerned with unreliable connectivity, reduced speeds and poor customer service.
Digging deeper into the survey results reveals a clear opportunity for the City to create a
successful City-owned broadband network (ref. Table 1 below). As shown in Table 1, survey
results show that Palo Alto’s population is very sophisticated when it comes to an understanding
of and the potential for a City-Owned FTTP network. Indeed, offering a service that provides
superior speed, reliability and service at a competitive cost was shown to be a sufficient and
compelling reason for support of such a City-wide initiative.
Key Findings City Provided Fiber Based Internet Opportunity
One third of households are dissatisfied with
their current internet services
Provide higher speed, affordable prices and more
reliability
Over half of the City has only one gigabit speed
broadband provider Create additional ISP choice for households
Price, speed and reliability are the most
important aspects of internet services
Provide a 100% fiber network with no data caps,
symmetrical speeds and better reliability at a
competitive price
Over 50% of households already subscribe to
internet service only
Provide high-quality, reliable and expandable services
to support streaming
Table 1: City of Palo Alto Population Key Findings
As highlighted in Magellan’s research, there are risks in developing a City-Owned FTTP
network. Unlike current City-Owned utility services, developing a City-Owned FTTP network or
City-Owned Internet Service Provider (ISP) would enter the CPAU into a “competitive business”
whereby the City would engage directly with 3rd party competitors whose actions the City has
limited ability to influence. As a service provider and as one of the few municipalities with a fully-
owned utilities service capability, however, the City of Palo Alto’s Utilities Department
(CPAU) is well-versed in managing a service organization that has provided quality services to
the citizens and businesses of Palo Alto since 1896. Given CPAU’s service reputation and
brand, as well as the survey results, it’s apparent the City’s population is confident in the CPAUs'
ability. But more to the point, they are demanding better pricing, higher speeds, and more
reliable broadband service. Given the opportunity to get all this AND keep dollars local to the
City of Palo Alto, creating a provider owned and operated by the City is a risk worth taking.
Page 3
6.a
Packet Pg. 81
At
t
a
c
h
m
e
n
t
:
A
t
t
a
c
h
m
e
n
t
A
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
(
1
4
7
4
2
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
)
Another risk deserving of discussion is the reaction by existing ISP providers in Palo Alto to the
introduction of a City-Owned ISP offering. It is entirely reasonable to expect a strong response to
the City’s initiative, likely through the offering of incentives for new customers or bonuses for
service extension by existing customers. Given the extent of the existing fiber network, as well
as its substantial service capabilities, the City is well positioned to offer faster broadband speeds
and lower monthly service costs while ensuring service reliability backed by the CPAU brand and
years of performance as a service business. As per Magellan Advisors suggested initial product
mix, the City would offer “synchronous” broadband services1 at both a higher speed and a lower
price. That is, CPAU could offer a very competitive service for both residential and business
particularly in terms of “speed-to-price” ratios as is highlighted in Table 2: Residential Service
Offering below.
Service Level Est. Monthly Rate (USD / Month)
5 Gigabit*USD 300
2 Gigabit*150
1 Gigabit*80
600 Megabit*65
300 Megabit*50
*Note: The City of Palo Alto’s service is based upon providing Synchronous Broadband access to subscribers, i.e.,
the same upload and download speeds.
Table 2: Residential Service Offering
An additional risk for a City owned and operated ISP is finding and hiring staff. Even before the
Covid-19 pandemic, CPAU struggled to fill vacancies, and for this reason Magellan Advisors was
asked to develop business plans and financial models based upon three assumptions with
respect to staffing models: 1) insourced; 2) outsourced; and 3) hybrid. A review of these
assumptions reveals that either a fully-outsourced model or a hybrid model could substantially
reduce the need for CPAU staffing whereas a fully insourced model would likely add to CPAU's
current staffing deficit. In reviewing Magellan’s output, both outsourced and hybrid models create
a viable business without overburdening the existing CPAU staff.
CPAU’s ability to create a viable business without overburdening the existing CPAU staff is
supported even when employing a very conservative Take Rate (how many Palo Alto
households are expected to subscribe to the City owned service). Under both the outsourced
and hybrid models, using conservative assumptions, it was demonstrated that a minimum Take
Rate needed for a City-Owned ISP’s revenues to exceed expenses is roughly 27% - 30% of
1 Synchronous service offers the same upload and download speeds whereas “asynchronous” connections offer faster download speeds and slower upload speeds and is the common and typical
service offered by AT&T and Xfinity in Palo Alto. Unlike asynchronous, synchronous connectivity comes
with guaranteed bandwidth services that are easy to manage and consistent in their speeds. If you pay for
a 500 Mbps connection, you get 500 Mbps upload and download speeds across your entire home or
office network.
Page 4
6.a
Packet Pg. 82
At
t
a
c
h
m
e
n
t
:
A
t
t
a
c
h
m
e
n
t
A
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
(
1
4
7
4
2
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
)
households. Interestingly, survey results indicate the City can comfortably assume a higher initial
Take Rate, so current financial modeling assumes a 40% Take Rate. Indeed, some survey
results indicate the Take Rate could be as high as 60%, but these have been discounted to a
more conservative assumption for the purposes of financial modeling. As such, it’s realistic to
assume the City’s Take Rate of 40% is highly likely and at least two of our business models are
realistic, fully outsourced and hybrid.
Finally, inherent in any technology investment is the risk of technology obsolescence. 5G has
recently been launched but remains a limited viable option, even within the City of Palo Alto.
Further and worthy of consideration, 5G and other mobile technology typically require a fiber
“backhaul” of data and traffic from the wireless antennae back to the telecom carrier’s central
office telecom switch. For future consideration, as 5G continues to roll out and as the City of
Palo Alto builds out its fiber network further, the need for wireless backhaul by service providers
may become another source of revenue for the City-owned fiber network. Satellite-based service
offerings are also available but may not be as compelling as a source of competition to a City-
owned ISP model given equipment costs of $599 for residential service and a high monthly
subscription fee ($110 per month).
Table 3 below summarizes key risks identified in our research and study.
RISK Recommendation
Competition CPAU would be entering a competitive business. Researching potential
competitor reaction to a City-Owned ISP recommended
Take Rate Assumptions Continuous monitoring and reporting of residential subscriptions
recommended
Technology Obsolescence
Current mobile service providers are beginning to offer 5G services. 5G
represents both a risk/threat as well as an opportunity. Researching the
potential impact of 5G services in Palo Alto recommended
Capital Investment
Current financial models represent a $117 million capital investment. $32
million is available immediately. Deficit of $85 million be funded via a
revenue bond. Monitoring of City’s Triple A credit rating recommended
Table 3: Summary of Key Risks
Why the time is right to build a City-Owned ISP in Palo Alto
As highlighted in Table 4 below, a City-owned and operated ISP would likely provide significant
value to Palo Alto residents. With more stable, synchronous broadband service at a reduced
price, together with CPAU’s high quality service, the City of Palo Alto population will find it easier
to work from home, to school from home and to access other services requiring stable,
high-speed synchronous broadband services. Looking out further, and considering the impact of
increased competition, it’s realistic to expect a projected 10% reduction in Internet spending
Page 5
6.a
Packet Pg. 83
At
t
a
c
h
m
e
n
t
:
A
t
t
a
c
h
m
e
n
t
A
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
(
1
4
7
4
2
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
)
community-wide amounting to $580 per residential household2. Importantly, the fees for this
service stay local with the City of Palo Alto, and dollars earned can be used to reinvest,
augment services, or reduce rates or offer additional service offerings.
Value Proposition Benefit
Improve reliability and speed Faster, more reliable internet makes it easier to work from home,
increases access to virtual learning and healthcare
Reduce community spend
Over 10 years, projected 10% reduction in internet rates could reduce
overall community-wide internet spending by $15 million or $580 per
residential household (25,876 households)
Equal access for all Next generation, high-speed internet available to all residents and
businesses in Palo Alto
Keep dollars local Fees for service stay local and dollars are reinvested to improve
broadband services or reduce rates
More control The community determines what services (immediate or future) are to be
provided by a City-Owned ISP
More choice Citizens have more broadband services options pushing providers to offer
better services and compete on price
Economic Tool Enhanced incentives for businesses to stay or relocate to the City
Table 4: Value Proposition
Any recommendation for providing a new service by the CPAU must recognize the strong
history of accomplishment and capability of our CPAU staff. As all are aware, the City of Palo
Alto Utilities Department currently manages all services associated with: electricity, water, sewer
& stormwater, gas and dark fiber. With a service history going back to 1896, the CPAU has a
strong “service first” mindset which has resulted in a strong brand and an appreciative
population. Additionally the CPAU has been recognized by the professional community for its
commitment to energy savings, sustainability, excellence, reliability and safety as is evidenced
by the long list of awards received, Ref.Appendix: Table 5 - CPAU Awards attached.
Finally, the CPAU has a demonstrated history of fiscal mindedness resulting in a well-run
organization with reserves appropriate for pushing into additional services, including FTTP. As
highlighted above, the City’s fiber reserve balance at the end of FY 2022 is approximately $34
million. These funds are immediately available to be used for operations and capital investment
related to the City of Palo Alto’s fiber network.3 Additionally, CPAU staff and Magellan Advisors
3 Meeting Sept. 28, 2015. City of Palo Alto staff interpretation of use restriction of fiber funds. Referencing
Part D of the motion, i.e., "wireless plans will not use fiber funds”. Sept 28, 2015 Action Minutes. Ref.:
2 According to the FY 2020 Utilities at a Glance, the City of Palo Alto has 25,876 residential and 3,973
commercial electric accounts. Source:
https://www.cityofpaloalto.org/Departments/Utilities/Customer-Service/Utilities-at-a-Glance
Page 6
6.a
Packet Pg. 84
At
t
a
c
h
m
e
n
t
:
A
t
t
a
c
h
m
e
n
t
A
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
(
1
4
7
4
2
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
)
confirmed that funding the capital investment shortfall of $85 million for building a City-Owned
ISP could come in the form of a “revenue bond” (also called municipal revenue bonds). Such a
bond is supported by the revenue from a specific project, such as a toll bridge, highway, or local
stadium. Revenue bonds that finance income-producing projects are thus secured by a specified
revenue source, in this case, revenues from the City-Owned ISP business. These differ from
general obligation bonds (GO bonds) that are repaid through a variety of tax sources. A key
takeaway is that revenue bonds are a class of municipal bonds issued to fund public projects
which then repay investors from the income created by that project. Unlike GO bonds, revenue
bonds are project-specific and are not funded by taxpayers. A second key takeaway is that a
decision to issue a revenue bond can come directly from the City Council and does not require a
ballot measure. More importantly, however, we understand that City staff believes that the
issuance of a fiber revenue bond will not impact the City of Palo Alto’s General Fund or other
utilities triple A ratings.
UAC Sub-Committee Recommendation
It is therefore our recommendation that the UAC recommend to the City of Palo Alto City Council
that the Council authorize CPAU to proceed forward with plans to offer fiber-based broadband
services through a City-Owned FTTP network and City-Owned Internet Service Provider to all
Palo Alto residents.
https://www.cityofpaloalto.org/files/assets/public/agendas-minutes-reports/agendas-minutes/city-council-a
gendas-minutes/00-archive/2015/09-28-15-approved-action-minutes.pdf
Page 7
6.a
Packet Pg. 85
At
t
a
c
h
m
e
n
t
:
A
t
t
a
c
h
m
e
n
t
A
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
(
1
4
7
4
2
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
)
Appendix: Table 5 - CPAU Awards
Date Organization Giving Award or
Recognition
Person/Program Receiving
Award or Recognition Title and Description of Award or Recognition
2021 American Public Power
Association (APPA)City of Palo Alto Utilities Smart Energy Provider Award
2021 U.S. Environmental Protection
Agency City of Palo Alto Utilities Champion Award for Responsible Appliance Disposal
Program
2021 American Public Power
Association (APPA)City of Palo Alto Utilities
Reliable Public Power Provider (RP3) for
industry-recognized leading practices in reliability &
safety
2019 American Public Power
Association (APPA)City of Palo Alto Utilities Smart Energy Provider Award
2019
American Public Power
Association (APPA)City of Palo Alto Utilities Energy Innovator Award for the Home Efficiency Genie
2019
American Public Power
Association (APPA)City of Palo Alto Utilities Reliable Public Power Provider -Utility designation for
providing reliable and safe service
2019 Arbor Day Foundation City of Palo Alto
Tree Line USA Utility - In recognition of quality tree care,
annual worker training, tree planting, and public
education
2018
National Research Center and
International City/County
Management Association
(ICMA)
City of Palo Alto Voice of the People Award for Excellence in Natural
Environment
2018 American Public Power
Association (APPA)City of Palo Alto Utilities
Reliable Public Power Provider (RP3) “Diamond” level –
the highest honor - for proficiency, sound business
practices, and a utility-wide commitment to safe and
reliable delivery of electricity, system improvement,
energy efficiency and workforce development.
2018 Arbor Day Foundation City of Palo Alto Utilities
Tree Line USA Utility – In recognition of quality tree
care, annual worker training, tree planting and public
education.
2017 Institute for Local Government City of Palo Alto 2017 Platinum Level Beacon Award Winner
2017 Smart Electric Power Alliance
(SEPA)City of Palo Alto Utilities
First place for Watts-per-Customer installed in 2016.
Fourth time on Top Ten list of utilities integrating the
most solar onto the grid.
2016 Moody's City of Palo Alto Water
Enterprise
Moody's upgraded the City's Water Enterprise bond
rating from Aa2 to Aa1, a rare event for water
operations.
2016 Solar Electric Power
Association City of Palo Alto Utilities Top Ten list of utilities that integrated the most solar into
the grid. #3 on the Watts-per-Customer list for 2015.
2016 Arbor Day Foundation City of Palo Alto Utilities
Tree Line USA Utility - In recognition of quality tree care,
annual worker training, tree planting, and public
education
2015 Arbor Day Foundation City of Palo Alto Utilities
Tree Line USA Utility - In recognition of quality tree care,
annual worker training, tree planting, and public
education
2015 Acterra City of Palo Alto Sustainability Award
(Ref.:https://www.cityofpaloalto.org/Departments/Utilities)
Page 8
6.a
Packet Pg. 86
At
t
a
c
h
m
e
n
t
:
A
t
t
a
c
h
m
e
n
t
A
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
(
1
4
7
4
2
:
C
o
l
l
e
a
g
u
e
s
M
e
m
o
)