Re: [WIP v2 2/2] config: include file if remote URL matches a glob

[Jonathan Tan <jonathantanmy@google.com>]

> > What's the easiest way to "try it and see", to add tooling and find out
> > whether the config files would be reopened during the second parse?
> > Because I suspect that we won't actually reopen those files, due to the
> > config cache.
> 
> strace -f?

Thanks - this might work.

> > So couldn't we do something like....
> >
> > pass #1:
> >  if (include)
> >    if (not hasRemoteUrl)
> >      open up path & parse
> >  put config into in-memory cache normally
> > pass #2: (and this pass would need to be added to repo_config() probably)
> >  if (include)
> >    if (hasRemoteUrl)
> >      open up path & parse
> >      insert in-order into in-memory cache
> >  don't touch existing configs otherwise
> >
> > I think it's in practice similar to the approach you're using (getting
> > around the weird ordering with a cache in memory), but we could reuse
> > the existing config cache rather than creating a new and different one.
> 
> I don't know enough to say if this two-step approach is better (although
> I'm slightly biased in that direction, since it seems simpler), but this
> just seems like premature optimization.
> 
> I.e. let's just read the files twice, they'll be in the OS's FS cache,
> which is unlikely to be a bottleneck for the amount of files involved.

OK - let me try this.

> That being said we do have exactly this cache already. See [1] and
> 3c8687a73ee (add `config_set` API for caching config-like files,
> 2014-07-28).
> 
> But I think that was added due to *very* frequent re-parsing of the
> entire config every time someone needed a config variable, not due to
> the I/O overhead (but I may be wrong).
> 
> So if we've got 100 config variables we need and 10 config files then
> 10*100 is probably starting to hurt, but if for whatever reason we
> needed 2*10 here that's probably no big deal, and in any case would only
> happen if this new include mechanism was in play.
> 
> 1. https://lore.kernel.org/git/1404631162-18556-1-git-send-email-tanayabh@gmail.com/ 

This might not work for the reasons I described in my reply to Emily
[1]. I'll try the read-twice version first.

[1] https://lore.kernel.org/git/20211109002255.1110653-1-jonathantanmy@google.com/