Using direct N-body simulations of self-gravitating systems we study the dependence of dynamical chaos on the system size N. We find that the N-body chaos quantified in terms of the largest Lyapunov exponent Λmax decreases with N. The values of its inverse (the so-called Lyapunov time tλ) are found to be smaller than the two-body collisional relaxation time but larger than the typical violent relaxation time, thus suggesting the existence of another collective time scale connected to many-body chaos.