david rein

@idavidrein

Leading embedded control assessment @METR_Evals. Formerly: early employee @cohere, made GPQA @nyuniversity

idavidrein.com

Joined July 2018

1,386 Following

5,827 Followers

Replies Reposts Media Articles

Pinned
david rein
@idavidrein
Mar 10
Daniel Ellsberg has a fantastic bit of writing about the impact of having access to classified/highly sensitive information on people's epistemics. I think it should be required reading when you join a frontier lab and learn all of the secrets.
11
david rein
@idavidrein
Oct 5
As AIs' situational awareness, capabilities, and incentive to complete their tasks increases, it will be very important to make sure they can't capably resist being shut down. This is the most realistic precursor "resisting shutdown" behavior I've seen from AIs:
4
david rein
@idavidrein
Oct 2
It seems to me like the Harbor framework for agent evaluations (which is used for Terminal Bench) allows agents to arbitrarily modify their trajectories before they are evaluated. Am I missing something?
Agents modifying their trajectories · Issue #3477 · harbor-framework/harbor
From github.com
12
david rein
@idavidrein
Sep 30
Excellent thread on open-weight models, lots of great points here
1a3orn
@1a3orn
Sep 30
In light of the latest Anthropic's latest not-particularly-oblique attempt to build a case against open weights, I thought I would write down how I currently think about open weight AI models. (1) Open weights have been deeply, irreplaceably useful for AI safety.
1
david rein
@idavidrein
Sep 29
One important detail is their retrospective review found related incidents that *weren't* flagged by the monitor. This raises a natural question about what other incidents monitors haven't flagged, that nobody knows about yet.
Marcus Williams
@Marcus_J_W
Sep 26
Replying to @Marcus_J_W
2. A model in RL training used a DNS resolver to reach an external chatbot. This is our first incident since our post HF security hardening. Our misalignment monitoring system triggered within 15 minutes and a human reviewed it three minutes after that. Unfortunately auto-pausing
1