Book contents
- Frontmatter
- Dedication
- Contents
- Preface
- Notation
- Contributors
- 1 Introduction to Information Theory and Data Science.
- 2 An Information-Theoretic Approach to Analog-to-Digital Compression
- 3 Compressed Sensing via Compression Codes
- 4 Information-Theoretic Bounds on Sketching
- 5 Sample Complexity Bounds for Dictionary Learning from Vector- and Tensor-Valued Data
- 6 Uncertainty Relations and Sparse Signal Recovery
- 7 Understanding Phase Transitions via Mutual Information and MMSE
- 8 Computing Choice: Learning Distributions over Permutations
- 9 Universal Clustering
- 10 Information-Theoretic Stability and Generalization
- 11 Information Bottleneck and Representation Learning
- 12 Fundamental Limits in Model Selection for Modern Data Analysis
- 13 Statistical Problems with Planted Structures: Information-Theoretical and Computational Limits
- 14 Distributed Statistical Inference with Compressed Data
- 15 Network Functional Compression
- 16 An Introductory Guide to Fano’s Inequality with Applications in Statistical Estimation
- Index
- References
2 - An Information-Theoretic Approach to Analog-to-Digital Compression
Published online by Cambridge University Press: 22 March 2021
Book contents
- Frontmatter
- Dedication
- Contents
- Preface
- Notation
- Contributors
- 1 Introduction to Information Theory and Data Science.
- 2 An Information-Theoretic Approach to Analog-to-Digital Compression
- 3 Compressed Sensing via Compression Codes
- 4 Information-Theoretic Bounds on Sketching
- 5 Sample Complexity Bounds for Dictionary Learning from Vector- and Tensor-Valued Data
- 6 Uncertainty Relations and Sparse Signal Recovery
- 7 Understanding Phase Transitions via Mutual Information and MMSE
- 8 Computing Choice: Learning Distributions over Permutations
- 9 Universal Clustering
- 10 Information-Theoretic Stability and Generalization
- 11 Information Bottleneck and Representation Learning
- 12 Fundamental Limits in Model Selection for Modern Data Analysis
- 13 Statistical Problems with Planted Structures: Information-Theoretical and Computational Limits
- 14 Distributed Statistical Inference with Compressed Data
- 15 Network Functional Compression
- 16 An Introductory Guide to Fano’s Inequality with Applications in Statistical Estimation
- Index
- References
Summary
Processing, storing, and communicating information that originates as an analog phenomenon involve conversion of the information to bits. This conversion can be described by the combined effect of sampling and quantization. The digital representation in this procedure is achieved by first sampling the analog signal so as to represent it by a set of discrete-time samples and then quantizing these samples to a finite number of bits. Traditionally, these two operations are considered separately. The sampler is designed to minimize information loss due to sampling based on prior assumptions about the continuous-time input. The quantizer is designed to represent the samples as accurately as possible, subject to the constraint on the number of bits that can be used in the representation. The goal of this chapter is to revisit this paradigm by considering the joint effect of these two operations and to illuminate the dependence between them.
Keywords
- Type
- Chapter
- Information
- Information-Theoretic Methods in Data Science , pp. 44 - 71Publisher: Cambridge University PressPrint publication year: 2021
References
Eldar, Y. C., Sampling theory: Beyond bandlimited systems. Cambridge University Press, 2015.Google Scholar
Gray, R. M. and Neuhoff, D. L., “Quantization,” IEEE Trans. Information Theory, vol. 44, no. 6, pp. 2325–2383, 1998.Google Scholar
Shannon, C. E., “Communication in the presence of noise,” IRE Trans. Information Theory, vol. 37, pp. 10–21, 1949.Google Scholar
Landau, H., “Sampling, data transmission, and the Nyquist rate,” Proc. IEEE, vol. 55, no. 10, pp. 1701–1706, 1967.Google Scholar
Shannon, C. E., “A mathematical theory of communication,” Bell System Technical J., vol. 27, pp. 379–423, 623–656, 1948.Google Scholar
Shannon, C. E., “Coding theorems for a discrete source with a fidelity criterion,” IRE National Convention Record, vol. 4, no. 1, pp. 142–163, 1959.Google Scholar
Berger, T., Rate-distortion theory: A mathematical basis for data compression. Prentice-Hall, 1971.Google Scholar
Walden, R., “Analog -to-digital converter survey and analysis,” IEEE J. Selected Areas in Communications, vol. 17, no. 4, pp. 539–550, 1999.Google Scholar
Candy, J., “A use of limit cycle oscillations to obtain robust analog-to-digital converters,” IEEE Trans. Communications, vol. 22, no. 3, pp. 298–305, 1974.Google Scholar
Oliver, B., Pierce, J., and Shannon, C., “The philosophy of PCM,” IRE Trans. Information Theory, vol. 36, no. 11, pp. 1324–1331, 1948.Google Scholar
Neuhoff, D. L. and Pradhan, S. S., “Information rates of densely sampled data: Distributed vector quantization and scalar quantization with transforms for Gaussian sources,” IEEE Trans. Information Theory, vol. 59, no. 9, pp. 5641–5664, 2013.Google Scholar
Matthews, M., “On the linear minimum-mean-squared-error estimation of an undersampled wide-sense stationary random process,” IEEE Trans. Signal Processing, vol. 48, no. 1, pp. 272–275, 2000.Google Scholar
Chan, D. and Donaldson, R., “Optimum pre- and postfiltering of sampled signals with application to pulse modulation and data compression systems,” IEEE Trans. Communication Technol., vol. 19, no. 2, pp. 141–157, 1971.Google Scholar
Dobrushin, R. and Tsybakov, B., “Information transmission with additional noise,” IRE Trans. Information Theory, vol. 8, no. 5, pp. 293–304, 1962.Google Scholar
Kolmogorov, A., “On the Shannon theory of information transmission in the case of continuous signals,” IRE Trans. Information Theory, vol. 2, no. 4, pp. 102–108, 1956.Google Scholar
Lapidoth, A., “On the role of mismatch in rate distortion theory,” IEEE Trans. Information Theory, vol. 43, no. 1, pp. 38–47, 1997.Google Scholar
Kontoyiannis, I. and Zamir, R., “Mismatched codebooks and the role of entropy coding in lossy data compression,” IEEE Trans. Information Theory, vol. 52, no. 5, pp. 1922–1938, 2006.Google Scholar
Zemanian, A. H., Distribution theory and transform analysis: An introduction to generalized functions, with applications. Courier Corporation, 1965.Google Scholar
Wolf, J. and Ziv, J., “Transmission of noisy information to a noisy receiver with minimum distortion,” IEEE Trans. Information Theory, vol. 16, no. 4, pp. 406–411, 1970.Google Scholar
Witsenhausen, H., “Indirect rate distortion problems,” IEEE Trans. Information Theory, vol. 26, no. 5, pp. 518–521, 1980.Google Scholar
Landau, H., “Necessary density conditions for sampling and interpolation of certain entire functions,” Acta Mathematica, vol. 117, no. 1, pp. 37–52, 1967.Google Scholar
Beurling, A. and Carleson, L., The collected works of Arne Beurling: Complex analysis. Birkhäuser, 1989, vol. 1.Google Scholar
Beutler, F. J., “Sampling theorems and bases in a Hilbert space,” Information and Control, vol. 4, nos. 2–3, pp. 97–117, 1961.Google Scholar
Kipnis, A., Eldar, Y. C., and Goldsmith, A. J., “Fundamental distortion limits of analog-to-digital compression,” IEEE Trans. Information Theory, vol. 64, no. 9, pp. 6013–6033, 2018.Google Scholar
Bennett, W., “Statistics of regenerative digital transmission,” Bell Labs Technical J., vol. 37, no. 6, pp. 1501–1542, 1958.Google Scholar
Kipnis, A., Goldsmith, A. J., and Eldar, Y. C., “The distortion rate function of cyclostationary Gaussian processes,” IEEE Trans. Information Theory, vol. 64, no. 5, pp. 3810–3824, 2018.Google Scholar
Kipnis, A., Goldsmith, A. J., Eldar, Y. C., and Weissman, T., “Distortion rate function of sub-Nyquist sampled Gaussian sources,” IEEE Trans. Information Theory, vol. 62, no. 1, pp. 401–429, 2016.Google Scholar
Kipnis, A., “Fundamental distortion limits of analog-to-digital compression,” Ph.D. dissertation, Stanford University, 2018.Google Scholar
Kipnis, A., Goldsmith, A. J., and Eldar, Y. C., “The distortion-rate function of sampled Wiener processes,” in IEEE Transactions on Information Theory, vol. 65, no.1, pp. 482-499, Jan. 2019. doi: 10.1109/TTT.2018.2878446Google Scholar
Kipnis, A., Reeves, G., and Eldar, Y. C., “Single letter formulas for quantized compressed sensing with Gaussian codebooks,,” in 2018 IEEE International Symposium on Information Theory (ISIT), 2018, pp. 71–75.Google Scholar
Kipnis, A., Reeves, G., Eldar, Y. C., and Goldsmith, A. J., “Compressed sensing under optimal quantization,,” in 2017 IEEE International Symposium on Information Theory (ISIT), 2017, pp. 2148–2152.Google Scholar