Trouble at Fermilab

[Vanadium 50]

Adrian Cho, who is always excellent, has a very good article on the recent difficulties Fermilab has been having: https://www.science.org/content/article/trouble-and-strife-deepen-famed-u-s-particle-physics-lab

A little background. In 2007, the current contractor received the DOE contract to run Fermilab. These are five-year (maybe its four) but if the lab does well on its "report card" it gets a one-year extension. Fermilab has stopped getting good grades, so they stopped getting those extensions, so the contract is being rebid.

A number of unhappy users have posted the following on the arXiv: https://arxiv.org/abs/2407.13924 Perhaps unsurprisingly, Fermilab management was angered by this. They have claimed that this is not all true (but I would say that it is not all false either).

 

[jedishrfu]

Its rare that employees and past employees band together to make a report like this.

I wonder if it will make a difference or will silent retaliation take place against those still there.

 

[jedishrfu]

We once went through something like this at our workplace. There was a new director and some discontent, so they brought an HR facilitator in to run some psychological games to discover workplace problems.

In the end, the report of action items to do was watered down to a trivial list that addressed only some of the significant problems of pay disparity. It was swayed somewhat by first-time employees who didn't realize how underpaid they were, nor how weak the educational benefits were, having never worked in the secular world.

Also, I'm sure the facilitator knew the outcome before it was produced and perhaps manipulated the data. Why would you run games when you can collect feedback, compile it together, and generate a report?

Also, the report was such that the new director could announce the outcome, make the changes without too much hassle, and look good doing it.

 

[Vanadium 50]

I don't think the ultimate outcome will be "business as usual". There will be a new contractor. There will be a Change of Administration in Washington. Now, that doesn't mean things will improve - just that they will change.

There are a number of stakeholders here, and the reports indicate that the relationships between them have grown increasingly toxic. Fermilab would argue that the bureaucratic overhead is exploding and is getting in the way of doing their job, and the DOE would counter that if you were doing your job, you wouldn't be getting so much attention and oversight. Neither side is exactly wrong, but its also not likely the best path forward.

 

[jedishrfu]

I had a boss that did that to me to protect his butt. I missed some dates because of unforeseen circumstances, and he then wanted me to sit down and write a detailed schedule of how I'd finish things. This took a few days and could have been more profitably spent doing the work.

I had to pad the schedule, guessing that there would be other unforeseen items. I basically doubled or tripled some items to get my schedule. I reviewed it with him, pared down some numbers, and got to work to get rid of him and this stupid, arbitrary schedule. He would then keep looking over my back until it got done.

It was a software project with other struggling programmers all being hounded to make a date in October. We would start fresh in January with a list of items to add for the new release:

- scope them in February,
- write specs in March,
- code in April,
- unit test in May,
- alpha in June,
- fix alpha stuff in July,
- beta in August,
- fix priority beta stuff in September

And deliver in October so that sales could sell it in November before the holidays.

We'd repeat this process each year to show that IBM teams could deliver new releases on a 12-month cycle instead of an 18-month cycle. The task and month were fuzzy targets as coding and testing could overlap, but alpha, beta, and release were hard targets.

I imagine Fermilab will go through this kind of audit with the new administration.

 

[Vanadium 50]

It is not clear that a new contarctor means new management. The new contractor may keep the Lab Director in place.

The arXiv document notes that Fermilab has had difficulty delivering beam. DOE wants proof they can resume so safely, and so "trust us, we have this" is not really an acceptable answer. There are components that are 50 years old, so "everything was OK last time" is also not totally reassuring.

 

[mfb]

Appendix E and H report problems that feel worse than difficulties with hardware.

 

[weirdoguy]

Appendix E and H report problems that feel worse than difficulties with hardware.

I was just about to say the same. It's disgusting.

 

[Vanadium 50]

I am not going to argue the truth or falsehood of E and H: Fermilab management quite obviously has a different opinion on exactly what happened.

I can say the experiments are struggling with Appendix E.

If FNAL employee A harasses employee B on site, its clear what their responsibilities are.
What if A and B are users?
What if the actions occurred off-site?
What if they are on different experiments?
What if only one is on a Fermilab experiment?
What if it's some non-harassment action - a user is convicted of assault after a bar fight?

The experiemnts are struggling to draw the line, and some have set up their own investigating and/or enforcement committees (often with little training). Where do you draw the line?

However, I would argue this is a problem on top of the difficulties Fermilab is having in meeting its mission.

 

[ohwilleke]

Does anyone have an idea what the timeframe is when this would have an impact on Fermilab operations?

I would imagine that even if the old Presidential administration and the new Presidential administration are in exact lockstep, that could interrupt dealing with the issue while the new Presidential administration gets is act together. The first half of the year following an inauguration is often something of a dead zone with respect to pending policy matters that weren't high profile enough to be discussed by the candidates during the campaign.

 

[Frabjous]

Does anyone have an idea what the timeframe is when this would have an impact on Fermilab operations?

I would imagine that even if the old Presidential administration and the new Presidential administration are in exact lockstep, that could interrupt dealing with the issue while the new Presidential administration gets is act together. The first half of the year following an inauguration is often something of a dead zone with respect to pending policy matters that weren't high profile enough to be discussed by the candidates during the campaign.

At the level of a Presidential Administration, Fermilab is a rounding error.

 

[Vanadium 50]

At the level of a Presidential Administration, Fermilab is a rounding error.

True, but CoAs are always disruptive.

But given that Fermilab isn't delivering beam at the moment, how much more disruptive could it get?

 

[ohwilleke]

At the level of a Presidential Administration, Fermilab is a rounding error.

The usual problem is just that. The decision is vested in someone three layers down from a cabinet secretary, which means that someone has to be appointed to all the positions above that post before that post can be filled.

 

[PeterDonis]

someone has to be appointed to all the positions above that post before that post can be filled.

That's generally not true. In fact the opposite is often true: there are vacant positions at or near the top, but the positions lower down are filled.

 

[phinds]

That's generally not true. In fact the opposite is often true: there are vacant positions at or near the top, but the positions lower down are filled.

 

[Vanadium 50]

There are three political appointees in the food chain - the head of the Office of Science, the Undersecretary for Science and the Secretary of Energy. That's it. In the event of vacancies, there is a procedure for temporarily filling them with career staff. (With some really crazy limits like 110 days),

The chaos during the CoA comes about because of something called the Chevron doctrine, which states that if the legislation is ambiguous, the agency is free to interpret any reasonable way the wish. This effectively let the agencies make decrees with the force of law. While the recent case of Lopez Bright reversed Chevron, arguing that legislation is a function of Congress, that doesn't make the existing legislation any less ambiguous, so the mess remains, at least for now. So the likely short-term impact will be more chaos, with some lawsuits thrown in.

The national labs are, by design, relatively immune to this.

In this case, we are supposed to hear who the new contractor is before the election (and thus before the CoA). I predict that a) the new contractor will have substantial overlap with the old contractor, changes to Fermilab management will be minor or zero, and business will go on as usual. I could be wrong, but that is my guess.

 

[Vanadium 50]

The transition to a new contractor starts tomorrow and is supposed to be complete by the end of the year. The new contractor has not been announced.

 

[Frabjous]

The transition to a new contractor starts tomorrow and is supposed to be complete by the end of the year. The new contractor has not been announced.

That’s on par for the federal government. It generally underestimates the time it takes lawyers to review the process to minimize the possibility of a successful protest.

 

[Vanadium 50]

They have a contractor, I am told. However, they have not announced who it is.

 

[Frabjous]

They have a contractor, I am told. However, they have not announced who it is.

The evaluation panel has probably scored the proposals. Where they are beyond that is an open question. The big question is how cost and organizational structure was handled in the call.

 

[Vanadium 50]

Again, I am told the decision is made but not announced. If you have evidence that something else is the case, please provide it.

 

[Vanadium 50]

The announcemeny has been made: Forward Discovery Group, LLC: the University of Chicago (UChicago), Universities Research Association, Inc. (URA), Amentum Environment & Energy, Inc. (Amentum), and Longenecker & Associates (L&A).

UChicago and URA are part of the present contractor, FRA, Germi Research Alliance.

Meet the new boss....

 

[Astronuc]

Meet the new boss....

Same as the old boss ... more or less?

Fermi Forward Discovery Group, LLC

Formed 9/26/23?

 

[Vanadium 50]

I don't anticipate huge changes with such a high degree of overlap in who is running things, no.

Amentum is a federal contractor that does all sorts of things, a spin-off of AECOM (who has done lots of work for the Labs). Longenecker & Associates is a bit more mysterious - I've been on DOE-OPA reviews with some of these people, and they seem to come from the NNSA side of the house, but I don't know for sure. Then again, I wouldn't, wouldn't I?

Formed 9/26/23?

Why would you need to create the entity if you were the unsucessful bidder?

Is this good news? Probably depends on what you thought should happen. If you thought no course correction or only a minor one was best, you are probably happy. If you though something more substantial needed to happen, you probbly aren't.

 

[Vanadium 50]

I am told 50 people were fired this week and perp-walked off site. Unofficial word from Management is this is all off it, unofficially nobody believes this.

Estimated beam delivery is half of what was planned,

 

[bob012345]

I think the most important issue is what’s going to happen to the Bison?