HomeMy WebLinkAboutStaff Report 2308-1933Item No. 3. Page 1 of 8
Utilities Advisory Commission
Staff Report
From: Dean Batchelor, Director Utilities
Lead Department: Utilities
Meeting Date: September 6, 2023
Staff Report: 2308-1933
TITLE
Overview of Palo Alto Fiber and Fiber Backbone Activities and Discussion of the Fiber Expansion
Plan
RECOMMENDATION
This memorandum and presentation are for discussion purposes only; no action is requested at
this time.
EXECUTIVE SUMMARY
On December 19, 2022, City Council directed staff to proceed with the Fiber Expansion Plan to
implement the Fiber Rebuild project and Phase 1 of the Fiber-to-the-Premises (FTTP) project. In
Phase 1, FTTP would be built out in selected areas of the city, and expanded gradually (Staff
Report ID # 14800)1. Construction of the fiber backbone and last mile infrastructure to provide
FTTP broadband internet to the community will be a significant undertaking for the City. Before
the City begins construction, staff needs to address the below items to control project costs,
minimize construction impacts to the community, and prevent major delays:
✓ Align fiber and grid modernization projects - The citywide engineering design for the electric
grid modernization project is currently underway. The grid modernization project will overlap
with the Fiber Expansion Plan in engineering make-ready work and construction. Staff will
analyze the projects to identify opportunities for alignment, which may help minimize costs and
reduce community disruptions, while trying to minimize impacts to the Fiber Expansion Plan’s
timeline.
✓ California Environmental Quality Act (CEQA) - The City must analyze and evaluate the
potential impacts of the project on various environmental factors and identify whether those
impacts can be mitigated.
1 Staf Report 14800 https://www.cityofpaloalto.org/files/assets/public/agendas-minutes-reports/agendas-
minutes/city-council-agendas-minutes/2022/20221219/20221219pccsm-amendedtime.pdf
Item No. 3. Page 2 of 8
✓ Coordination with AT&T - The existing joint pole agreement with AT&T requires
coordination between the City and AT&T to relocate 3rd party telecom equipment on utility poles
and provide space for new fiber attachments, while remaining in compliance with California
Public Utilities Commission General Order 95 (GO 95) requirements for overhead electric line
construction to insure electric utility service and secure personnel safety.
PROJECT DESCRIPTION
The proposed Fiber Expansion Plan is to construct a new fiber optic backbone and implement
Phase 1 of fiber-to-the-premises (FTTP) infrastructure (passing ~ 20% of homes and businesses)
to provide communication services throughout the City of Palo Alto. However, the proposed plan
will be dynamic in nature as efforts to align grid modernization and fiber projects increase in
priority and impact the designs and timelines. For informational purposes, the initial plan for fiber
construction includes installation of approximately 83 miles of fiber optic cables, consisting of
about 48 miles of below-ground cable installations and 35 miles of aerial cable installations using
~6,000 existing utility poles; one fiber hut (20’ x 40’) for networking equipment; underground
utility vaults (36” x 48” x 34”); and above ground utility cabinets (17” x 17” x 36”).
BACKGROUND
City Council considered three courses of action for a City-owned FTTP service and directed staff
to proceed with the phased build approach. Under this approach, FTTP will be built out in
selected areas of the city using $34 million from the Fiber Fund and $13 million from the Electric
Fund, and the project will expand gradually from there to eliminate the need for debt financing.
Building the fiber backbone and last-mile infrastructure to provide FTTP broadband internet to
the community will require significant planning, coordination, communication and construction
over the next 18 – 24 months. With the electric grid modernization project approved in the FY
2024 budget, staff needs to determine how to align the grid modernization project and projects
under the Fiber Expansion Plan to help minimize utility engineering pole make-ready work, pole
replacements, noise disruption, and construction activity in neighborhoods.
On May 1, 2023, City Council approved Amendment Number 4 to Contract Number C20176363
with Magellan for Fiber program management, organizational change management, network
operations and technical support, and utility pole electric make-ready engineering (Staff Report
#2303-1215)2. Although the phased build approach to FTTP will decrease the City’s financial risk
and increase Council’s control over the velocity of the buildout, the City does not have the
inhouse staffing to fully pursue the fiber backbone project and first phase of FTTP while
coordinating with electrification grid modernization efforts. These projects require significant
staffing and specialized skill sets over the next few years. While it is economically prudent to
2 Staff Report 2303-1215 Item Number 8
https://cityofpaloalto.primegov.com/Portal/Meeting?meetingTemplateId=1455
Item No. 3. Page 3 of 8
utilize available staffing resources, the City must also strategically invest in external resources for
a successful roll-out.
On May 30, 2023, the Finance Committee reviewed the FY 2024 Budget during the budget
approval process. The FY 2024 Budget included a new Capital Improvement Project, the Grid
Modernization for Electrification Project (EL-24000).
On June 19, 2023, the City Council approved the FY 2024 budget with the new Grid Modernization
for Electrification Project. The approval of the electrification project accelerated efforts to align
electrification and fiber construction, which has begun to impact the Fiber Expansion Plans.
ANALYSIS
Align fiber and grid modernization projects
The City recently conducted an electrification study to evaluate the impacts of projected
electrification loads on Palo Alto’s distribution and substation transformers, primary/secondary
distribution circuits, and to propose upgrades needed to mitigate overloads. The estimated cost
to construct the necessary electric system upgrades for a 100% electrification scenario is
between $220 million to $306 million, depending on the approach. Specifically, the grid
modernization project involves replacing and upgrading over a thousand single phase pole top
transformers. Approximately 300,000 circuit feet of open wire secondary conductors in the
distribution system will be replaced with aluminum aerial cable. The workplan for this project is
a staged, multi-year approached designed to accomplish the upgrade of the electric distribution
system to meet the City’s goal of being ready for full-scale electrification by 2030.
Currently, staff is in the process of implementing a pilot project (~600 customers) to convert a
4kV area to 12 kV and upgrade the transformers and secondary networks to facilitate
electrification. This work is expected to be completed by the beginning of 2024, depending on
other factors which impact project timelines, such as material lead times, pole make ready, and
alignment with the fiber projects. The purpose of the pilot project is to determine the feasibility
of electrification design and construction methods, explore opportunities to align with fiber
expansion, and implement best practices to facilitate the most cost-effective deployment of
resources.
After the pilot project, the first four years of the electrification project will make approximately
70% of the electric system ready to accept full-scale electrification by residents. This will be
accomplished by upgrading over one hundred miles of the overhead electric system that serves
these residents and upgrading substation transformers to meet the increased demand. After the
overhead electric system is upgraded, the City will upgrade underground residential systems for
additional capacity with tentative completion by 2030.
Given the hundreds of miles of overhead and underground construction activity between FTTP
and electric grid modernization, staff is exploring various construction phasing options to
minimize disruption, construction activity, and construction costs within neighborhoods while
Item No. 3. Page 4 of 8
avoiding prolonged deployments. In addition to construction constraints, staff does not have
construction for both projects in parallel.
Staff is currently engaged in discussions with an electric engineering consultant to assess the
feasibility of utilizing the fiber engineering make-ready work previously prepared by Magellan for
the grid modernization project. If feasible, the City of Palo Alto Utilities (CPAU) may be able to
expand the pilot area of the grid modernization, accelerate the design, and align construction of
grid modernization with phase one of FTTP. Staff will be analyzing the cost savings, timeline, and
resources (staffing and contractors) required for project alignment.
California Environmental Quality Act (CEQA)
The preliminary assessment, or Initial Study, will evaluate the project’s potential impacts on
various environmental factors and identify whether those impacts can be mitigated. If the
project’s impacts can be mitigated to a less-than-significant level, then the City may issue a
Mitigated Negative Declaration (MND) or Negative Declaration (ND). If the preliminary
assessment of the project determines one or more significant impacts on the environment
despite mitigation, a full environmental impact report (EIR) will be required, which involves a
more comprehensive and detailed analysis of the project’s impacts and alternatives. CPAU is
seeking to retain a CEQA consultant to prepare a CEQA Initial Study in accordance with local,
state, and federal statutes. The Initial Study identifies physical, biological, social and economic
factors that might be affected by the fiber expansion project. Some environmental factors that
could potentially be affected by this project include aesthetics, air quality, geology/soils,
hazardous materials, noise, utilities services, and hydrology/water quality. Specific examples of
likely CEQA analysis include:
o Hydrological analysis to see if any proposed fiber lines would run through areas near bodies
of water and show that there would not be significant impacts to waters of the state or waters
of the US, as well as provide recommendations for how to protect nearby waterbodies even if
there are no direct impacts (e.g. through proper stormwater protection measures)
o Air quality analysis (including understanding how to run claimed calcs for the proposed
project to evaluate air quality emissions and GHG emissions)
o Noise modeling (including analysis of potential impacts on nearby sensitive receptors from
construction noise and vibrations)
o Hazardous materials analysis (including evaluation and mitigation if necessary, of potential
impacts to construction workers and sensitive receptors especially if any of the dark fiber runs
through areas of the city runs through a plume)
Coordination with AT&T
CPAU currently has a long-standing joint pole agreement with AT&T executed in 1918. The
current joint pole agreement and pole intent process (including billing) may not be feasible to
accommodate the increased volume of pole transactions expected with the grid modernization
Item No. 3. Page 5 of 8
and Fiber Expansion Plan. This issue will likely cause significant delays in both projects, so CPAU
is reevaluating membership in the Northern California Joint Pole Association (NCJPA). The NCJPA
is a 100+ year old non-profit organization formed to support members and provide standardized
cost-sharing methodologies for utility poles. Members have “joint pole equity” and voluntarily
share expenses for pole ownership, maintenance, use, setting, replacement, dismantling,
relinquishment or removal of jointly owned poles. Municipalities that join the NCJPA are required
follow “NCJPA rules” for their pole assets. In the past, CPAU operated without membership with
the NCJPA because the volume of pole replacements was manageable by staff and there were
additional costs associated with becoming a member. Given the anticipated increase in pole
work, NCJPA membership may be necessary to ensure pole work related to the long-term fiber
and electrification projects is conducted in a methodical, timely and cost-efficient manner. CPAU
may need to add resources to manage the volume of pole make-ready work and replacements.
Other Updates
Make-Ready Engineering
Magellan completed the survey of approximately 6,000 utility poles in Palo Alto. The survey
identified the current load and condition of each pole in preparation for the new fiber backbone
and FTTP. The survey included detailed fielding and walk-out of all routes to validate running
lines, existing utilities, and constructability. For aerial construction, Magellan provided
makeready engineering pole data to determine pole preparation and pole replacement, as well
as timeframes for the work to be completed. For the new fiber backbone and phase one of FTTP,
CPAU will be passing 1750 existing poles, attaching to 1300 poles, replacing 100 poles, and
coordinating with third parties to move their telecom equipment lower on 325 poles. In addition
to the fiber expansion project, CPAU has a recurring Wood Pole Replacement capital
improvement project to replace approximately 100 deteriorated wood poles annually. Wood
poles are used to support overhead utility lines throughout the City of Palo Alto. Though poles
are inspected, tested, and treated to maintain their integrity, over time poles will deteriorate to
the point that they are no longer in compliance with GO 95 requirements or safe for community
and utility workers.
Fiber Hut Sites:
Fiber huts are usually a prefabricated building (10’ x 20’) containing electronics and network
equipment that connects fiber to neighborhoods. The hut also has backup generators and HVAC
systems to maintain equipment within its operating environmental specifications. Two fiber huts
are recommended for the citywide network for Palo Alto Fiber. Staff and Magellan evaluated
potential fiber hut sites and considered their proximity to strategic areas, planning/land use
requirements, and existing infrastructure. As a result, staff narrowed down the potential
locations to the Colorado substation and anticipated space at City Hall. The Colorado substation
was previously identified as an ideal location due to its proximity to the area it serves and its
current use for utilities. City Hall is anticipated to have vacated space after the Police Department
moves to the new Public Safety Building. City Hall would be an ideal location for a secondary hut.
Item No. 3. Page 6 of 8
Due to the centralized location and potential square footage, this site may also be considered for
a future data center, further securing the future of the City‘s data infrastructure.
Equinix/Data Center:
Equinix is currently the City’s data center provider. They are a vendor-neutral multitenant data
center provider where numerous internet service providers and content delivery networks can
interconnect and exchange internet traffic between their networks. As the City expands its fiber
footprint, more data center space will be needed. For FTTP, the City will have to increase the
number of leasing cabinets (vertical racks sited next to each other, each capable of holding
multiple termination/switching boxes in stacked configurations) from five to 12 cabinets. The
City is undergoing discussions with Equinix to lease additional space available in their Palo Alto
location and to explore other Equinix locations. The City may also consider increasing the size
of the data center at City Hall.
Invitation for Bid (IFB) Construction Package:
If opportunities exist to align FTTP and grid modernization, CPAU and Magellan may gather and
consolidate all supporting documents to bid out construction of FTTP and grid modernization.
The IFB package will include construction-ready prints for fiber and power, construction details,
splice details, pole make-ready details, construction standards as well as all documentation
needed to complete the bid package. This shall include a written summary and scope of work,
estimated costs to construct, as well as labor estimates, bill of materials, and the vendor list.
Invitation for Bid (IFB) Construction Package:
If opportunities exist to align FTTP and grid modernization, CPAU and Magellan may gather and
consolidate all supporting documents to bid out construction of FTTP and grid modernization.
The IFB package will include construction-ready prints for fiber and power, construction details,
splice details, pole make-ready details, construction standards as well as all documentation
needed to complete the bid package. This shall include a written summary and scope of work,
estimated costs to construct, as well as labor estimates, bill of materials, and the vendor list.
Staffing:
As the City solidifies staffing plans in parallel with contracted services, the Director of
Information Technology will take on the responsibilities of an Assistant Director for Palo Alto
Fiber on an interim basis. In the FY 2024 adopted budget, four (4) new FTE positions for the
dark fiber expansion and implementation of FTTP were created. These positions will be
recruited and filled as needed during the various stages of the project.
•Assistant Director - To provide the vision and strategy for the new fiber business and
lead a high-performance team to quickly execute. Responsibilities include and are not
limited to: overseeing fiber enterprise fund budget timelines and milestones (including
the dark fiber optic business), managing the roll out and expansion of the fiber optic
network, Internet Service Provider (ISP) operations, customer service operations, and
business development.
Item No. 3. Page 7 of 8
•Manager Utilities Telecommunications / "Outside Plant Manager" - To lead the
construction process, installation and repair service technicians, and ensure the
outside plant processes run efficiently and smoothly. Responsibilities include and are
not limited to: overseeing construction, managing engineers and contractors, achieving
construction budget timelines and milestones.
•Manager Information Technology / "Sr. Network Engineer/Architect” – To lead the
development process of the overall architecture of the broadband system and
manage the system. Responsibilities include and are not limited to: managing the
network and supporting technical needs across the organization.
•Manager Utilities Telecommunications / "Sales and Marketing Manager" - To
develop the marketing strategy and lead marketing initiatives. Responsibilities
include and are not limited to: meeting revenue objectives, cultivating strong
partnerships, and maintaining retention rates.
.
FISCAL/RESOURCE IMPACT
This report is for informational purposes so there is no resource impact. Based on Council and
UAC input, staff will return with specific actions associated with efforts to align the fiber
expansion and electric grid modernization projects.
STAKEHOLDER ENGAGEMENT
On November 2, 2022, the UAC unanimously recommended to build fiber backbone and FTTP
under a phased approach with existing funds ($34 million from Fiber and $13 million from
Electric). The UAC expressed the goal of FTTP is to provide ubiquitous or citywide high-speed
internet access to all residents and businesses in Palo Alto. If financially self-sustaining, and
deemed successful, the first phase of FTTP can become a springboard to a citywide FTTP
deployment within five years.
On December 19, 2022, City Council directed staff to proceed with the Fiber Expansion Plan to
implement the Fiber Rebuild project and Phase 1 of the Fiber-to-the-Premises (FTTP) project
without debt financing. Included in the Council’s motion was direction to: a) maximize number
of homes and businesses passed; b) consider promotional rates to increase take rate; c) define
leading indicators and metrics to determine success; and d) recommend future Council action
to accelerate expansion if metrics are positive, including a potential bond to streamline
construction and compress construction time as much as feasible.
ENVIRONMENTAL REVIEW
The UAC’s discussion of the fiber expansion project and its potential alignment with the grid
electrification project is not a project requiring California Environmental Quality Act review,
because it is an administrative governmental activity which will not cause a direct or indirect
physical change in the environment.
Item No. 3. Page 8 of 8
ATTACHMENTS
Attachment A: Presentation
AUTHOR/TITLE:
Dave Yuan, Strategic Business Manager