Can I start collecting my ex-spouse at my FRA while still working, then switch to my account later

NO - for a couple of reasons:

1.  the primary account holder has to be already retired and receiving benefits

2.  and even if he were drawing benefits you would not be able to do it at your FRA 

because you would be under the “deemed filing” rules.

SSA.gov - Filing Rules for Retirement and Spouses Benefits  

Basically it says that when you file for Retirement benefits, you are filing for all of the Retirement benefits to which you are entitled.

Retirement benefits and Spousal Benefits are the same type of SS benefit - Retirement

So when you file at your FRA, you would be entitled to either your own benefits or your Spousal benefit - so you would be filing for either of these - SSA will give you the one that pays the most.  

And the same change also created the rule that the primary account holder has to be already retired for a spousal benefit to be given.

@SusanO457775 I agree with Gail1's reply. The Bipartisan Budget Act of 2015 (see link in Gail1's reply) changed SS law regarding filing for Retirement/Old Age Benefits and Spousal Benefits. The above law kept the incentives for delaying SS Benefits, but it eliminated folks from receiving one type of benefit (i.e., Spousal) while at the same time earning a bonus for delaying the other benefit (i.e., retirement/old age).

As you may not be aware, you can receive SS Benefits (i.e., retirement/old age) at your Full Retirement Age (FRA) and continue to work. You do not need to stop working.Moreover, your initial SS Benefit at FRA will be reviewed each year and may be increased after your additional Covered Earnings are posted to your Covered Earnings Record. Most folks confuse their Covered Earnings Record with having an account with the SS Program. We do not have an account with the SS Program.

You did not indicate if your amount of SS Benefits at FRA exceeds your amount of Spousal Benefits (as a divorced spouse) or vice versa.So, it is not clear what your intent is. If your are in search of additional income, you may have been awarded pursuant to your Dissolution of Marriage a portion of your marital assets (including pensions, 401 K, IRA,etc) you and your ex-spouse accumulated during 24 years of marriage. If so, a Qualified Domestic Relations Order (QDRO) may have been entered to provide you with access to such benefits at FRA or even earlier. 

As I indicated above, we do not have an account with the SS Program. We do not have a contractual guarantee or any equity interest in the SS Program. So, SS Benefits are not governed by a QDRO. However, some QDROs entered after the Bipartisan Budget Act of 2015 may address  SS Spousal Benefits for a divorced spouse by dividing marital assets accordingly.