Consider switching to the Time series visualization type. It is a more capable and performant version of this panel.
Migrate
Some features are not supported in the new panel yet.